Auditing AI models for Verified Deployment under Semantic Specifications
Authors: Homanga Bharadhwaj
De-An Huang
Chaowei Xiao
Anima Anandkumar
Animesh Garg
Venue: ICLR 2022 Submitted (2022)
Keywords: auditing deep learning
verification
interpretability

Abstract: Auditing trained deep learning (DL) models prior to deployment is vital for preventing unintended consequences. One of the biggest challenges in auditing is in understanding how we can obtain human-interpretable specifications that are directly useful to the end-user. We address this challenge through a sequence of semantically-aligned unit tests, where each unit test verifies whether a predefined specification (e.g., accuracy over 95\%) is satisfied with respect to controlled and semantically aligned variations in the input space (e.g., in face recognition, the angle relative to the camera). We perform these unit tests by directly verifying the semantically aligned variations in an interpretable latent space of a generative model by building a bridge with the DL model. Our framework, AuditAI, bridges the gap between interpretable formal verification and scalability. With evaluations on four different datasets, covering images of chest X-rays, human faces, ImageNet classes, and towers, we show how AuditAI allows us to obtain controlled variations for verification and certified training. We address the limitations of the standard approach of verifying using only pixel-space perturbations.

AUDITING AI MODELS FOR VERIFIED DEPLOYMENT UNDER SEMANTIC SPECIFICATIONS
Anonymous authors Paper under double-bling review
ABSTRACT
Auditing trained deep learning (DL) models prior to deployment is vital for prevent- ing unintended consequences. One of the biggest challenges auditing is the lack of
specifications for the DL models that are directly useful to the auditor. We address this challenge through a sequence of semantically-alignee unit tests, where each unit test verifies whether predefined specification (e.g., accuracy over 95%) is satisfied with respect to controlled and semantically aligned variations in the input space (e.g.. in face recognition, the angle relative to the cam- era). We enable such unit tests through variations in a semantically inntrreeally-ttterrtaabllle latent space of generative model. Further, conduct certified training for the DL model through shared latent space representation with the generative model. With evaluations on four different datasets, covering images of chest X-rays, hu- man faces, ImageNet classes, and towers, we show how AuditAI allows us to obtain controlled variations for certified training. Thus, our framework, AuditA bridges the gap between semantically-alin aliggned formal verification and scalability. http:///siee..oogllooooo wwwgoogoooooooooooee.eeeeeeoeee. ciwiee/aut.aaiiii
INTRODUCTION
Deep learning (DL) models are now ubiquitously deployed in a number of real-world applications, many of which are safety critical such as autonomous driving and healthcare (Kendall et al., 2019; Miotto et al., 2018: Senior al., 2020). As these models are prone to failure, especially under domain shifts, it is important to know when and how they are likely to fail before their deployment, process we refer to as auditing. Inspired by the failure-mode and effects analysis (FMEA) for control systems and software systems (Teng & Ho, 1996), we propose to audit DL models through a sequence of unit tests, where each unit test verifies whether a pre-defined specification (e.g.. accuracy over 95%) is satisfied with respect to controlled and semantically meaningful variations in the input space (e.g., the angle relative to the camera for a face image). Being semantically- aligned is critical for these unit tests to be useful for the auditor of the system to plan the model's deployment.
semantically-alige
The main challenge for auditing DL models through semantically-aligned unit tests is that the current large-scale DL models mostly lack interpretable structure. This makes it difficult to quantify how the output varies given controlled semantically-aligee that aim to bring interpretable formal verification DL models (Henriksen & Lomuscio, 2020: Liu et al., 2019), the scale is still far from the millions if not billions of parameters used in contemporary models (He al., 2016; landola et al., 2014; Brown et al., 2020).
input variations. While there are works
On the other hand, auditing has taken the form of verified adversarial robustness for DL mod- els (Samangouei et al., 2018; Xiao et al., 2018a; Cohen et al., 2019). However, this has mostly focused on adversarial perturbations in the pixel space. for example, through Interval Bound Propaga- tion (IBP), where the output guaranteed to be invariant to input pixel perturbations with respect to Lp norm (Gowal et al., 2018; Zhang et al., 2019b). While these approaches are much more scalable to modern DL architectures, the pixel space variations are not monantically-alignee meaning they do not directly relate to semantic changes in the image, unlike in formal verification. Consider unit test that verifies against the angle relative to the camera for a face image. A small variation in the angle (e.g., facing directly at the camera versus 50 to the left) can induce a large variation in the pixel space. Current certified training methods are far from being able to provide guarantees with respect to such large variations in the pixel space with reasonable accuracy.

---

TRAINING
VERIFICATION
DEPLOYMENT
TEST
RESULT
Model Spec-sheet
Trained Model
Certified
Model
Unit Test
95%
fd
Training
Data with labels
Unit Test
86%
Yes
Output y
Specification
Unit Test
92%
not satisfied
glasses
Check Range
Do not use the model for input
Controlledgeneration
Inputx
Large unsupervised dataset
Generative mode bridge
Figure I: Basic outline of AuditAI. The training phase involves the design and training of the algorithms for a particular use case. The auditor lists a set of specifications that the trained model must satisfy. The auditor receives the trained model from the designer and with the set of specifications has to determine whether the model satisfies the specifications. they updates. After that, the model deployed, with model spec-sheet detailing the verified range of operation. At deployment, latent embeddings of the input inform whether the input is within model-speccifications. Based on the spec-sheet, it can be decided in which settings to use the model.
satisfied, then the model is sent back the designer for further
Our Approach. In order to overcome the above limitations, we develop a framework for auditing, AaditAI. We consider typical machine learning production pipeline (Fig. 1) with three stages, the design and training of the model, its verification, and finally deployment. The verification is crucial determining whether the model satisfies the necessary specifications before deployment.
We address the gap between scalability and interpretability by proposing to verify specifications for variations directly in a semantically-alignee latent space of a generative model. For example in Fig. 1, unit test verifies whether given face classification model maintains over 95% accuracy when the face angle is within d", while unit test 2 checks under what lighting condition the model has over 86% accuracy. Once the verification is done, the auditor then the verified specification to determine whether to use the trained DL model during deployment.
latent variations, create bridge between the generative model and the DL model such that they share the same latent space. We incorporate a variant of IBP (Zhang al., 2019b) for verification with respect to perturbations in the latent space. Further this leads to a tighter and a much more practical bound in the output space compared to pixel-based certified training. We also show that AuditA, tAI can verify whether a unit test is satisfied by generating a proof for verification based on bound propagation. Fig. gives an overview of our auditing framework and Fig. 2 elaborates on the generative model bridge.
For semantically-
Summary of Contributions:
1. We develop a framework, AuditAI for auditing deep learning models by creating a bridge with generative model such that they share the same semantically-aligne latent space.
We propose unit tests as semantically-alignes way to quantify specifications that can audited.
3. We show how IBP can be applied to latent space variations, to provide certifications of semantically- aligned specifications.
We show that Audi tAT is applicable to training, verification, and deployment across diverse datasets: ImageNet (Deng et al., 2009), Chest X-Rays (Irvin al., 2019; Wang al., 2017), LSUN (Yu et al., 2015), and Flicker Faces HQ (FFHQ) (Karras et al., 2019). For ImageNet, we show that Aud tAT can train verifiably robust models which can tolerate 25% larger pixel-space variations compared to pixel-based certified-training counterparts for the same overall verified accuracy of 88%. The variations are measured as L2 distances in the pixel-space. The respective % increase in pixel- space variations that can be certified for Chest X-Rays, LSUN, and FFHQ are 22%, 19%, 24%. We conclude with human-study of the quality of the generative model for different ranges of latent-space variations revealing that pixel-space variations up 62% of the nominal values result in realistic generated images indistinguishable from real images by humans.
2

---

2 AUDITAI: A DEEP LEARNING AUDIT FRAMEWORK
In this section, we describe the details of our framework, AuditAI outlined in Fig. for verifying and testing deep models before deployment. We propose to use unit tests to verify variations of interest that semantically-aligeee are given by several semantically aligned specifications for the end-user to follow during deployment. In the following sub-sections, we formally define unit tests, outline the verification approach, and describe the specifics of AuditAI with a GAN-bridge shown in Fig. 2.
The advantage of Audit is that the verified input ranges
2.1 UNIT TEST DEFINITION
Consider a machine learning model f : + y that predicts outputs y E y from inputs x E X' in dataset D. Each of our unit test can be formulated as providing guarantees such that:
(x, 5 Vx s.t. .c(22) E St,in: y = f(x)
(1)
Here, i subscripts an unit test, encoder e, extracts the variation of interest from the input and St denotes the set of range of variation that this unit test verifies. If the condition is satisfied, then our unit test specifies that the output of the model (xx would satisfy the constraint given by F(:.)) O.
For example, x could be an image of a human face, and f could be a classifier for whether the person is wearing eyeglasses (f I 0 means wearing eyeglasses and f 0 means not wearing eyeglasses). Then (-()) could be extracting the angle of the face from the image, and St,in could be the set {x)) Vo [=x(=] < 30"] constraining the rotation to be smaller than 30'. And F(x, y)
-s(xx) (xrg=) < 0 says that our classifier output would not be changed from the corresponding output of zge, the face image of without any rotation. In this case, when the end-user is going to deploy f on a face image, they can first apply e.(.) to see if the face angle lies within St,in to decide whether to use the model f.
2.2 VERIFICATION OUTLINE
Given specification F(x, y) O, we need next answer the following questions:
1. How do we obtain the corresponding components (e,, St,n) in Eq. (1)?
2. What can do guarantee that Fxr y) 0 is indeed satisfied in this case?
For the sake of illustration, we first consider a scenario with a less realistic assumption. We will then discuss how can relax this assumption. In this scenario, we assume that we are already given the encoder e; and a corresponding generator gi that inverts e, can extract the face angle of given face image. On the other hand, gi (Zge, d²) would be able to synthesize arbitrary face image za- that is the same as Zye except being rotated by do.
Continuing the previous example,
Given the generator Ji and the en- coder e,, we propose to obtain Sr,nn building on interval bound propaga- tion (IBP) (Gowal et al., 2018). We include a treatment of IBP prelimi- maries in Appendix I. Here, Sn is the largest set of variations such that the specification F(r,y) is satisfied. Given a set of varia- tions St,n IBP-based methods can be used to obtain a bound on the out- put of the network Stout
angle
Certified Training
Eyagissos
0
Expression
Check Specdtiction
x
7
Elig Foge
{y ly S y S. yy} and it can be checked whether the specification F(x, y) I 0 is satisfied for all values in Stout-
Figure 2: Instead of variations the level of input (e.g. pixels). consider variations in the latent space. Given specification, we a variant interval-houne propagation (IBP) to verify the spec- ification is satisfied for e-hall latent space variations for particular latent dimensions.
We can start with a initial set Sp
and apply gs to this set. Then, we can first find out what would be the corresponding convex bound of variations of SD in the image space X. We can subsequently propagate this bound in X' through f to y' to get Syo and check whether F(x,9) v) < 0 is satisfied. By iteratively searching for the largest set Si such that F(x, y) S 0 is still satisfied by Si Sioutt we can obtain St,n given F.
3

---

2.3 CERTIFIED TRAINING THROUGH LATENT REPRESENTATION
In the previous section, we are able to find the set Si,in such that Eq. (1) is satisfied given specification F. encoder e,, generator J. However, there are several challenges that limit the practical application of this scenario. First and foremost, while significant progress has been made in generative models, especially controlled generation (Brock al., 2018; Karras et al., 2017), the assumption of having 9i apriori is still unrealistic. In addition, since the propagated bound is convex, one can imagine that the bound in X (high dimensional image-space) would be much larger than needed. This leads to a much narrower estimate of in While it could be true that Eq. (1) is satisfied, we still need non-trivia size of Sinin for the framework to be useful. For example, if St,in constrains face angle rotations to be within 1. then it may not be practical for the end-user.
For auditing ML models thoroughly, we require clear separation between the development and testing phases (i.e. black-box verification). However, it might also be desirable in practice to train the model in such a way that the unit tests are likely to be satisfied in the first place. In this case, we do have a tighter connection between the designer and the verifier. We show that by relaxing the black-box constraint, we can simultancously address the previously listed challenges in a white-box setting.
Now that we have access to the internal working and design of the ML model f, we propose to bypass the image generation step in our verification process through the use of latent disentangled
representations. More specifically, consider a ML model mapping f X - Z + y (denote by fa 2 + y the downstream task) and generative model mapping 9 : X' - 2 + X, such that f and g share common latent space Z by virtue of having common encoder e(-) (Fig. 2).
Since St,in is a subset of the range of e, we have Si,in S Z. This further implies that we do not have to apply IBP through y and the whole f. Instead we only have to apply it through fa. which does not involve the expansion to the pixel space X. We show in the experiment (Section 4) that this is important to have practically useful Sn In addition, AuditAl also alleviates the need of having a perfect generative model, as long as we can learn a latent space Z through an inference mechanism.
Learning the latent space. There are three requirements for the latent space. First, it should be ammantically-aalgnedd for us to specify unit tests in this space. A good example is disentangled representation (Higgins et al., 2018; Tran et al., 2017), where set of dimensions of the latent code would correspond to a semantically aligned variation of the original image. As shown in Figure 2, a latent dimension could correspond to the pose variation, and we can select e; to be the value of this dimension, and S,tn as the proper range that the model would be verified for. Second, our model should be verifiable, where the latent space is learned such that the corresponding St,n as large as possible given a specification function F. In this case. Audite tAI can be seen as doing certified training (Zhang et al., 2019b) to improve the verifiability of the model. Finally, the latent space should be able to perform well on the downstream task through fa- Combining these three criteria, our full training criteria: C Lusk + LLape + ôggen combines three losses Lyen- Lipees and Lusk that would encourage interpretability through the latent space of generative model, verifiability, and task performance respectively. y and 5 are relative weights (between O and 1) for the loss function terms. We explain each of the losses below:
Verifiability (Lspec). Let fa be a feedforward model (can have fully connected / convolutional / residual layers) with K layers z (W+= + by), k = O..... K - 1, where, 2p e(x), žς denotes the output logits, and h2 is the activation function for the 1.th layer. The set of variations
S,,in could be such that the 7, norm for the 5th latent dimension is bounded by i.e. St.n {=
|| - zo,illp < }). We can bound the output of the network Sout = {ax IK S Zk {uk} through a variant of interval bound propagation (IBP). Let the specification be F(z,y) = c z + d <
0, Vz E St,,, zk- fa(z).
For a classification task, this specification implies that the output logits 2 should satisfy a linear relationship for each latent space variation such that the classification outcome arg max, =k.i remains equal to the true outcome Ymne corresponding to 2g. To verify the specification, IBP based methods search for counter-eet counter-example, example, which in our framework amounts to solving the following optimization problem, and checking whether the optimal value is < 0.
max F(z, v) cT zk + d s.t. 24+1 = (W + bR) k = 0, ..., K...K..... - zES,,
By following the IBP equations for bound guund-rrpaggaton, we can obtain upper and lower bounds [EK, =k] which can be used to compute the worst case logit difference =k,y EK Mone between the
4

---

true class Viue and any other class y. We define 2g kK.y (if y + Mlue); 2K 2m. (otherwise). Then, define Lsgee Lspce CE(2K: &K Mrne). In practice, we do not use IBP, but an improved approach CROWN-IBE (Zhang ct al., 2019b) that provides tighter bounds and more stable training,
Jime
based on the AutoLiRPA library (Xu et al., 2020). We mention details about this in the Appendix.
Task Performance (Lusk)) Let, CE(-) denote the standard cross-entropy loss. Lusk captures the overall accuracy of the downstream task and can be expressed as Lusk CE(2K, Yuve).
Generative Model (Lgen) ). Finally, we have a loss term for the generative model, which could be VAE, GAN, variants of these. If the model is GAN, then we would have optimization objective for GAN inversion. An assumption is being able to learn a disentangled latent space with semantically aligned concepts. Let the overall loss be Lgen which could also encapsulate other losses for more semantically-alignee disentanglement. Typically we would train this generative model on a large amount of unlabelled data that is not available for training the classifier we are auditing.
2.4 DEPLOYMENT
Based the perturbation radius and the nominal 2n, we have different ranges [=] - 11,,, =8 for the verified error at the end of training the classifier. Here, zi e(x") V n = I....N in the dataset Drain and 1, for the 7th dimension (corresponding to the 7th unit test) and is 0 otherwise. We can convert this per-sample bound to a global bound for the 7th latent code, [z zil by considering an aggregate statistic of all the ranges, for example
+ [
[2,,2]] = [p(tzū, 1=,) of - 1, 1) + 1,e], + (f + +,1))=1))
Here, ju(-) denotes the mean and (-) denotes the standard deviation. Note that this is a global bound on the range of variations for only the latent dimension corresponding to the unit test, and does not concern the other dimensions. During deployment, the end-user receives the model spec-sheet containing values of [E,. for each unit test and results with different values of perturbation radius e. Now, if the end-user wants to deploy the model on an evaluation input Teval, they can encode it to the latent Z space to get code Aeval and check whether Seval, E [5o,s Zn,dl in order to decide whether to deploy the model for this evaluation input.
3 THEORETICAL RESULTS
We show that although AuditAl considers perturbations in the latent space that allows a large range of semantic variations in the pixel-space, it retains the theoretical guarantees of provable verification analogous to IBP-based adversarial robustness (Zhang et al., 2019b; Xu et al., 2020).
Theorem 1 (Verification. The auditor can verify whether the trained model from the designer satisfies e(x) be the latent
the specifications. by generating proof of the same). Let E X be the input, code, fa be a ferd-forward neural network with bounded derivatives, and Sy.in {z : || - lll S }} be the set of latent perturbations. Then, the auditor is guaranteed to be able to genenate proof for verifying whether the linear specification on output logits cf 2k + d S Vz E Stii 2K fd(z) (corresponding to unit test i) is satisfied.
Proof. Here we provide a proof sketch and mention details in the Appendix A.2. The verifier can generate proof by searching for worst-case violation max.eS,... c² 2ạ + d s.l. Zk+1 h + by) k = O,.... K - 1. If the optimal value of this < 0, then the specification is satisfied. The verifier applies Interval- bound Propagation (IBP) to obtain an upper bound 2 on the solution of this optimization problem, and checks whether the upper bound - < O.
4 EXPERIMENTS
We perform experiments with four datasets in different application domains. We choose the datasets ImageNet, LSUN, FFHQ, and CheXpert where current state of the art generative models have shown strong generation results (Karras et al., 2019; 2017; Brock et al., 2018). We show an illustration of what the generated images under latent perturbations look like in Fig 3. Through experiments, we aim to understand the following:
1. How do the verified error rates vary with the magnitude of latent perturbations?
2. How does AuditAI compare against eexx-pperturbation based verification approaches?
How does AuditAI perform during deployment under domain shift?
5

---

4.1 IMAGENET
ImageNet contains 1000 different classes of ob- jects. We consider a BigBiGAN model (Don- ahue & Simonyan, 2019) as the base genera- tive model such that the latent space class conditional, and we can obtain latent codes cor- responding to specific classes. So, latent per- turbations correspond to variations in different images within the same class. The specification for verification is that the classifier outputs the correct class for all latent perturbations e. This an unit test because we can verify with respect to each class separately. Table 1 shows results for different e on the test set. The results con- firm the intuition that increasing increases the verified error due to wider latent perturbations.
Figure 3: Generated samples from latent space manip- ulation of features on the ImageNet, CheXpert, LSUN Tower, and FFHQ datasets. We respectively show two images of hen looking left and right, chest X-ray images with and without pneumonia, towers with and without vegetation and faces, with and without glasses. These are examples of features that can be controlled through latent space manipulations.
In Table 2, we compare a pixel-based robust training approach with our framework, both trained with the same underlying algorithm CROWN- IBP (Zhang et al., 2019b). For the same verified test error, we have the corresponding latent perturbation for AuditAl tAI (call it {1) and the pixel perturbation for the pixel-based approach (call e2). Based on E2 compute the bound for the intermediate latent node, 621 Since comparing 61 and 621 directly may not be meaningful due to different nominal values of the latent code, we consider the fraction //nom and plot the resulting values in Table 2. We observe that 21/znom >> 621 Znomg: indicating tolerance to much larger latent variations. Since a wider latent variation corresponds to a wider set of generated images, AuditAI effectively verifies over a larger range of image variations, compared to perturbations at the level of pixels. In addition, when we translate the latent perturbations of Audi tAI the pixel space of the generative model, we obtain on average 20%, and 17% larger variations measured as Lg distances compared to the pixel-based approach for a verified error of 0.88 and 0.50 respectively.
Table I: Verified Error fraction on a subset (100 classes) of the ImageNet dataset. Results are on the held-out test set of the dataset and correspond to training the classifier with the same e during training. S.D. is over four random seeds of training. 100 classes is the average value of 100 unit tests over all the 100 classes while min class and max class respectively correspond the minimum and maximum verified errors for unit tests over each class. Alongside each we represent the % with respect to the nominal value it corresponds to (the nominal value of e is 0.02, so = 0.005 is 25% of 2.0). Lower is better.
ImageNet = 0. 0.005(25%% (501)
(751)
0.02(100%)
0.05(200%)
100 classes
0.20=0.05
0.51
0.57
0.89
0.94
min class
0.07
0.33+0.02
0.41=0005
0.58
0.86+0007
max class
0.41+00.0
0.68
0.73+00006
0.93+00055
0.99++0.05
Table 2: Comparison of pixel-based variation for CROWN-IBP and Audi tAT on 100 classes of the ImageNet dataset. We tabulate the values below relative the nominal values of the corresponding latent codes (i.e.
ef Znom) and observe that 21/znom >> 621 //aamgg of generated images. AuditAI effectively verifies over a larger range of image variations. Higher is better.
wider latent perturbation corresponds to a wider set
Verified Error
0.88
0.50
Method
Pixel-based E21 AuditAI e Pixel-based E21
AuditAl E1
% Latent variation 0.21%
20%
0.14%
12%
4.2 CHEXPERT
CheXpert (Irvin et al., 2019) contains 200,000 chest radiograph images with 14 labeled observations corresponding conditions like pneumonia, fracture, edema etc. We consider PGGAN (Karras et al., 2017) as the base generative model such that the latent space is conditonnii. We consider three separate binary classification tasks with respect the presence absence of three conditions:

---

pneumonia, fracture, and edema. We consider the entire dataset for training the generative model, but for the classifiers we only use positive (presence of the condition) and negative (absence of the condition) data with an 80-20 train/test split. Table 3 shows results for different on the test set for each classifier.
As sanity check, in order to determine whether the generative model behaves correctly with respect to class conditional generation, we consider an off-the-shelf classifier for pneumonia detection trained on real images of pneumonia patients (Rajpurkar et al., 2017). We randomly sample 1000 images from the generative model for each value of e = 0.5, 1.0, 1.5, 2.0 and evaluate the accuracy of the off- the-shelf classifier on these generated images. We obtain respective accuracies 85%, 84%, 81 %, 75% indicating highly reliable generated images.
Table 3: Verified Error fraction the CheXpert dataset. Results the held-out test set of the dataset and correspond to training the classifier with the same during training. S.D. over four random seeds of training. Each row is an unit test as it corresponds to verifying classifier particular condition. Alongside each e we represent the % with respect to the nominal value it corresponds to (the nominal value of is 2.0, so = 0.5 is 25% of 2.0). Lower is better.
Unit Test 6 (25%)
= 1. 150(5
= 1.5(7%%) = 2.0(100%) 4.0(200%)
Pneumonia 0.02:10.01
0.12:10.03
0.24
0.43=#0.04
0.51+00.5
Edema
0.03++0003
0.14++0.04
0.22++0.04
0.41++0.03
0.55++0.06
Fracture
0.04++0.02
0.14++0.03
0.22++0.05
0.42++0.02
0.52++0.03
Deployment under domain-shift. In order to better understand the process of deploying the trained model on dataset different from the one used in training, we consider the dataset NIH Chest X- rays (Wang et al., 2017). This dataset contains (image, label) pairs for different anomalies including pneumonia and edema. We perform certified training on CheXpert for different values of (results of this are in Table 3) and deploy the trained model to the end-user. The end-user samples 5000 images from the NIH Chest X-rays dataset, encodes them the latent space with trained encoder E(.) and uses the trained downstream task classifier fa(:) to predict the anomaly in the image. In Table 4, we list the precision, recall, and overall accuracy among the 5000 images for two conditions, presence of pneumonia and edema. We observe that the results for AuditA IAI are on average 5% higher than the pixel-based approach. This intuitively makes sense because from Table 2 and section 4.1, we see that Audi LAI covers about 20% larger variations in the pixel space and so the odds of the images at deployment lying in the verified range are higher for AuditAI.
4.3 FFHQ
Flicker Faces HQ (Karras et al., 2019) contains 70,000 unlabeled images of normal people (and not just celebrities opposed to the CelebA (Liu et al., 2015) dataset). We train StyleGAN model (Karras et al., 2019) on this dataset and use existing techniques (Bau et al., 2020a; Shen & Zhou, 2020; Shen al., 2020) identify directions of semantic change in the latent space. We identify latent codes (set of dimensions of Z) corresponding to expression (smiles le), pose, and presence of eyeglasses. We define the task for the classifier be detection of whether eyeglasses are present and define unit tests to be variations in the latent code with respect to expression and pose (while keeping the code for eyeglasses invariant). Table shows the results of these unit tests corresponding to different latent perturbations e. We see that the verified error is up to 20% for over 50% larger variations compared to nominal values. Based on this spec-shect table, during deployment, the end-user can check whether the encoding of input image lies in the desired accuracy ranges for pose and expression, and then decide whether to use the model.
4.4 LSUN TOWER
The LSUN Tower dataset (Yu et al., 2015) contains 50,000 images of towers in multiple loca- tions. Similar to the FFHQ setup above, we train StyleGAN model on this dataset and use existing techniques (Shen & Zhou, 2020; Zhu et al., 2020) to identify directions of semantic change in the latent space. We identify latent codes corresponding to cl clouds, vegetat ion, and sunlightt http///rgggg........eoes.ess We define the task for the classifier to be detection of whether vegetation is present and define unit tests to be variations in the latent code with respect to how cloudy the sky is, and how bright the scene is (while keeping the code for vegetation invariant). Table shows the results of these unit tests corresponding different latent perturbations c. We see that the
7

---

Table 4: Domain shift study. Evaluation on the NIH Chest X-Rays dataset, after being trained on the CheXpert dataset. We perform certified training of the models with e 1.0 on CheXpert and deploy it on 5000 images of a different dataset quantity evaluation performance under potential domain shitt. We tabulate the values of precision, recall, and overall accuracy. Higher better.
Method
Pixel-based
AuditAI Recall
Metric
Precision
Recall
Accuracy
Precision
Accuracy
Pneumonia 0.78=0.03
0.80=#0.03 0.74+0.04
0.83++0.04 0.84
0.84 0.78=0005 0.79=0.03
Edema
0.79=0.05
0.01 0.75+0.03
0.86+00.4
Table 5: Verified Error fraction on the FFHQ and LSUN datasets. For FFHQ. the task to classify whether eycglasses are present, and the unit tests correspond to variations with respect to pose and expression of the face. For LSUN Tower, the task is to classify whether green vegetation is present on the tower, and the unit tests correspond to variations with respect to clouds in the sky and brightness of the scene. Results lest set of the datasel and correspond to training the classifier with the same during training. S.D. is four random seeds of training. Lower is better.
the hcld-out
FFHQ
6 0.5(25%) 6 1.0(50%) 6 1.5(75%) 6 2.0(100%) c = 3.0(150%)
Pose test
0.01+00002 0.01=00022
0.20+0005
0.37=0003
0.48
0.53=0.06
Expression test
0.13+0.04
0.29=0.04
0.58+0001
0.56=0.03
LSUN Tower
c 0.5(25%)
<= (=1.0(50%)
<= 1.5(75%)
e 2.0(100%)
c= 3.0(150%)
Cloud test
0.02.1:.03
0.14:10.02
0.32
0.44:10.06 0.49+-0.04
0.51=0.03
Brightness test 0.01+-0.01
0.18+-0.03
0.35++0.05
0.54
verified error is up to 18% for over 50% larger variations compared to nominal values. Based on this spec-sheet table, during deployment, the end-user can check whether the encoding of input image lies in the desired accuracy ranges for clouds in the sky and brightness of the scene, and then decide whether to use the model.
4.5 ATTACKING IMAGES AUDITED BY AUD ITAI IN THE PIXEL-SPACE
In this section, we analyze what fraction of images audited by AuditAI can be attacked in the pixel-space through adversarial corruptions. We create test examples with varying brightness of the scene, corresponding to the setting in Table 5. These are real variations directly by modifying pixels to changing the brightness of the scene We evaluate Audi t trained the same way as in Table 5, but evaluated on this test set after certified training. For e 0.5, 96 out of 1000 images (only 9.6%) certified by AuditAI can be attacked in the pixel-space, while for e = 1.5, 85 out of 1000 images (only 8.5%) certified by AuditA: can be attacked. We use CROWN-IBP (Zhang et al., 2019b) to determine if an image is attackable in the pixel-space. These results show that the certification provided by AuditA is robust, and although verification is done in the latent space of generative model, the results hold under real mextt-perturbations as well.
5 HUMAN STUDY: HOW REALISTIC ARE THE GENERATED IMAGES?
We employ Amazon Mechanical Turk (Buhrmester et al., 2016) to better understand the generative model trained for the Chest X-ray dataset and determine the limits of latent space variations for which generations are realistic. This study is to determine the maximum value of e for which generations are realistic to the human eye, such that we verify and report in the spec-sheet values of e only up to the range of realistic generations. We note that this study is to understand the generative model and not the trained classifier. We employ similar experimental protocol as (Isola et al., 2017: Zhang et al., 2016) and mention details in the Appendix A.4. We generated 2000 images by sampling latent codes within each e range of variation. Alongside each e we represent the with respect to the nominal value it corresponds to (the nominal value of is 2.0, so c 0.5 is 25% of 2.0) in Fig 4. We see that the humans who participated in our study had close to 50% correct guesses for e up to 1.0, indicating generations indistinguishable from real images. For 70% indicating that the generations in these ranges of latent variations are not realistic, and so the verified accuracy for these ranges should not be included in the spec-sheet.
4.0 the correct guess rate is above
8

---

6
RELATED WORK
Mechanical Turk study results
100
1811
Multiple prior works have studied a subset of the audit problem in terms of adversarial robust- ness (Raghunathan et al., 2018a; Dvijotham al.). Adversarial examples are small pertur- bations to the input (for images at the level of pixels; for text the level of words) which mis- lead the deep learning model. Generating ad- versarial examples has been widely studied (Xu al., 2018; Zhang et al., 2019a) and recent ap- proaches have devised defenses against adver- sarial examples (Xie et al., 2020; Zhang al., 2019b). Some of these approaches like (Gowal al., 2018) and (Zhang al., 2019b) pro- vide provable verification bounds on an out- put specification based on the magnitude of perturbations in the input. These approaches have largely showed robustness and verifica- tion to norm bounded adversarial perturbations without trying to verify with respect to more semantically aligned properties of the input. Some papers (Mohapatra et al., 2020; Ruoss et al., 2020; Balunović et al., 2019; Wong & Kolter, 2020) consider semantic perturbations like rotation, translation, occlusion, brightness change etc. directly in the pixel space, the range of semantic variations that can be considered are more limited. This is a distinction with AuditAL, where by perturbing latent codes directly (as opposed to pixels), the range of semantic variations captured are much larger.
80
1450
1150
60
990
1060
0.5 (25%) 1 (50%)
2 (100%) (200%)
(300%)
latent variation E
Figure 4: Amazon MTurk study involving 2000 human evaluations for each value of in study of CheXpert Chest X-ray images. We showed pairs of real and gen- crated images to the MTurk workers who chose to take part in our study and asked them to identify which image is real and which is fake (generated). A value close 50% indicates that the MTTrr workers were unable to distinguish the real images from the generated ones, in- dicating that the generated images looked as good as the real images.
related notion to auditing is expalainable AI (Gunning, 2017). We note that auditing is different from this problem because explanations be incorrect while remaining useful. Whereas in auditing, our aim is not to come up with plausible explanations for why the model behaves in a certain way, but quantify how behaves under semantically-alignel variations of the input.
Some papers (Mandelbaum & Weinshall, 2017; Van Amersfoort et al., 2020) have used model confidence and ucnertainty estimates for rejecting out-of-distribution samples at test time, which is form of auditing. In (Mitchell et al.. 2019) and (Gebru et al., 2018), the authors proposed checklist of questions that should be answered while relcasing DL models and datasets that relate to their motivation, intended use cases, training details, and ethical considerations. These papers formed a part of our motivation and provided guidance in formalizing the audit problem for DL models. We provide detailed survey of related works in section A.6.
DISCUSSION AND CONCLUSION
In this paper developed framework for auditing of deep learning (DL) models. There are increasingly growing concerns about innate biases in the DL models that are deployed in a wide range of settings and there have been multiple news articles about the necessity for auditing DL models prior to deployment" 2 Our framework formalizes this audit problem which we believe is step towards increasing safety and ethical use of DI. models during deployment.
One of the limitations of AuditAI is that its interpretability is limited by that of the built-in generative model. While exciting progress has been made for generative models, we believe is important to incorporate domain expertise to mitigate potential dataset biases and human error in both training & deployment. Currently, AuditAI doesn't 't directly integrate human domain experts in the auditing pipeline, but indirectly uses domain expertise in the curation of the dataset used for creating the generative model. Although we have demonstrated AuditAl primarily for auditing computer vision classification models, we hope that this would pave the way for more sophisticated domain-dependent AI Al-auditing tools & frameworks in language modelling and decision-making applications.
http:///wwwacccioooooooooo/////p Jul 10, 2019 questions-remaind, Feb 11, 2021

---

REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. International Conference on Machine Learning (ICML), 2018.
Mislav Balumović, Maximilian Baader, Gagandeep Singh, Timon Gehr, and Martin Vechev. Certifying geometric robustness of neural networks. Advances in Neural Information Processing Systems 32, 2019.
David Bau, Hendrik Strobelt, William Peebles, Bolei Zhou, Jun-Yan Zhu, Antonio Torralba, et al. Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727, 2020a.
David Bau, Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units deep neural network. Proceedings of the National Academy of Sciences. 11744)::0071-33007 2020b.
David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba. Paint by word. arXiv preprint arXiv:2 10951, 2021.
Andrew Brock, Jeff Donahue, and Karen Simonyan, Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.109% 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14 165, 2020.
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow: Thermometer encoding: One hot way to resist adversarial examples. International Conference on Learning Representations, 2018.
Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. Amazon's mechanical turk: A new source of inexpensive, yet high-quality data? 2016.
Nicholas Carlini and David Wagner. Adversarial examples are easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 3-14. ACM, 2017a.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 38th IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017b.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: (w..//02..0077 06705, 2019.
Hongge Chen, Huan Zhang, Pin- Yu Chen, Jinfeng Yi, and Cho-Jui Hsieh. Attacking visual language grounding with adversarial examples: A case study on neural image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2587-25555 2018
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. International Conference on Machine Learning, pp. 1320. PMLR, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-24482 255.. Icee, 2009.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in Neural Information Processing Systems, pp. 10541-1055/ 2019.
Krishnamurthy Dj Dvijotham, Robert Stanforth, Sven Gowal, Chongli Qin, Soham De, and Pushmeet Kohli. Efficient neural network verification with exactness characterization.
10

---

Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1625-1634, 2018.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III and Kate Crawford. Datasheets for datasets. arXiv preprint arXiv: 1e0::00010011 2018.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness interval bound propagation for training verifiably robust models. arXiv preprint 1rXiv:1810.127155 2018.
David Gunning. Explainable artificial intelligence (xai). Defense Advanced Research Projects Agency (DARPA), nd Web, 2(2), 2017.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In ICLR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Patrick Henriksen and Alessio Lomuscio. Efficient neural network verification via adaptive refinement and adversarial search. In ECAI 2020, pp. 2513-2520. IOS Press, 2020.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint 1812v: 2018.
Forrest landola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv: 1404. 2014.
1404 1.04.180 1869,
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Hleus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, al. Chexpert: A large chest. radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 590-597, 2019.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1125-11344 2017.
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. Certified robustness to adversarial word substitutions. arXiv preprint arXiv: (009:/990.00000 00986, 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 100. 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 International Conference on Robotics and Automation (ICRA), pp. 2244.,25.. IEEE, 2019.
Changliu Liu, Tomer Amon, Christopher Lazarus, Christopher Strong, Clark Barrett, and Mykel J Kochenderfer. Algorithms for verifying deep neural networks. arXiv preprint 1003: 2019.
11

---

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Michael E Houle, Grant Schoenebeck, Dawn Song, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference Learning Representations (ICLR), 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations, 2018.
Amit Mandelbaum and Daphna Weinshall. restance-baseds confidence score for neural network classifiers. arXiv preprint UrXiv:170..09444 2017.
Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley. Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics, 19(6):133-11266 2018.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model eards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pp. 220-229, 2019.
Jeet Mohapatra, Tsui-War Wei Weng, Pin- Yu Chen, Sijia Liu, and Luca Daniel. Towards verifying robustness of neural networks against a family of semantic perturbations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 244-252 2020.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium Security and Privacy (SP), pp. 582-597. IEEE, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0002. 2021.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversar- ial examples. International Conference on Learning Representations (ICLR), arXiv preprint 1801v 093444, 2018a.
Aditi Raghunathan, Jacob Steinhardt, and Percy S Liang. Semidefinite relaxations for certifying robustness to adversarial examples. In Advances in Neural Information Processing Systems, pp. 10900-10910, 2018b.
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. Chexnet: Radiologistlleve pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225, 2017.
Anian Ruoss, Maximilian Baader, Mislav Balunović, and Martin Vechev. Efficient certification of spatial robustness. arXiv preprini arXiv:2009.099 09318, 2020.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-GAN: Protecting classifiers against adversarial attacks using generative models. arXiv preprint 1rx5vv 2018.
Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Zidek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792):70--71, 2020.
Yujun Shen and Bolei Zhou Closed-fom factorization of latent semantics in gans. arXiv preprint arXiv:2007.06600. 2020.
Yujun Shen, Ceyuan Yang, Xiaoou Tang. and Bolei Zhou. Interfacegan: Interpreting the disentan- gled face representation learned by gans. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
12

---

Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin Vechev. Fast and effective robustness certification. In Advances in Neural Information Processing Systems, pp- 10825-10836, 2018.
Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. Robustness certification with refinement. ICLR, 2019.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. arXiv preprint 1710. 10766, 2017.
Sheng-Hsien Gary Teng and Shin- Yann Michael Ho. Failure mode and effects analysis. International journal of quality & reliability management, 1996.
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for inssiinvariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pp. 1415-1424, 2017.
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In International Conference Machine Learning, pp. 9690-9700. PMLR, 2020.
Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. Efficient formal safety analysis of neural networks. In Advances in Neural Information Processing Systems, pp. 6369- 6379, 2018.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on wcakly-supervised classi- fication and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017.
Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Duane Boning, Inderjit Dhillon, and Luca Daniel. Towards fast computation of certified robustmess for ReLU networks. In International Conference on Machine Learning, 2018.
Eric Wong and J Zico Kolter. Learning perturbation sets for robust machine learning. arXiv preprint arXiv:2007.08450, 2020.
Chaowei Xiao, Ruizhi Deng. Bo Li, Fisher Yu, Mingyan Liu, and Dawn Song. Characterizing adversarial examples based on spatial consistency information for semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 217-234, 2018a.
Chaowei Xiao, Bo Li, Jun- Yan Zhu, Warren Hc, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. IJCA118, 2018b.
Chaowei Xiao, Jun- Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. (C,, 2018c.
Chaowei Xiao, Ruizhi Deng. Bo Li, Taesung Lee, Benjamin Edwards, Jinfeng Yi, Dawn Song, Mingyan Liu, and Ian Molloy. Advit: Adversarial frames identifier based on temporal consistency in videos. In Proceedings of the IEEE International Conference Computer Vision, pp. 3968- 3977, 2019a.
Chaowei Xiao, Dawei Yang. Bo Li, Jia Deng, and Mingyan Liu. Meshadv: Adversarial meshes for visual recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6898-6907, 2019b.
Cihang Xie. Mingxing Tan, Boqing Gong. Alan Yuille, and Quoc Le. Smooth adversarial training. arXiv preprint arXiv:2006. 14536, 2020.
Kaidi Xu, Sijia Liu, Pu Zhao. Pin- Yu Chen, Huan Zhang, Quanfu Fan, Deniz Erdogmus. Yanzhi Wang, and Xue Lin. Structured adversarial attack: Towards general implementation and better interpretability. arXiv preprint arXiv: I 1808 01664, 2018.
13

---

Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai- Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Processing Systems, 33, 2020.
Chih-Kuan Ych, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 150.: 03365, 2015.
Huan Zhang, Tsui-Wei Weng, Pin-Yu Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions. In Advances in Neural Information Processing Systems (NIPS), 2018.
Huan Zhang, Hongge Chen, Zhao Song, Duane Boning, Inderjit S Dhillon, and Cho-Jui Hsich. The limitations of adversarial training and the blind-spot attack. ICLR, 2019a.
Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsich. Towards stable and efficient training of verifiably robust neural networks. arXiv preprint arXiv:1906.066 2019b.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pp. 649-666. Springer, 2016.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. arXiv preprint arXiv:1710.11320 2017.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European Conference on Computer Vision, pp. 592-608. Springer, 2020.
14

---

A APPENDIX
A.1 PRELIMINARIES: INTERVAL-BOUND PROPAGATION
In this section, provide preliminary treatment of Interval-Bound Propagation based robust training and verification. Let f be a feedforward classifier model that outputs one of N classes. The input the network is an image 2o, and it has certain ground truth class Vime- The classifier is trained to minimize a certain misclassification loss. The verification problem is to check whether there is some set S,n (2r0) around perturbations of zo such that the classification output of the model remains invariant from that of Yauee
Let f have K layers 2k+1 hy + bx), k O..... 1, such that 20 E (mn())). 2k denotes the output logits, and he is the activation function for the htth layer. If we consider the set of variations to be Smn(1)) = {æ|l|æ - solloo I c). then we want to verify the specification (no( We can write this as general linear specification
that arg max; ²K,i Unne
cT 2 + d < 0 V 20 E (zu)
To verify the specification, IBP based methods search for a counter- exammeer--xmppee that violates the specifi- cation constraint, by solving the following optimization problem and checking whether the optimal value
max c' + d s.t. 2x+1 = (W + bx) k = 0, K -1 ZES..
Since solving this optimization problem exactly is NP-hard (Zhang et al., 2019b), IBP finds an upper bound on the optimal value and checks whether the upper bound < 0. To get upper bound on 2K, we can propagate bounds for each activate 23 through axis-aligned bounding boxes, (c) S 24, S (E, where i indexes dimension. Based on this, the specification is upper- bounded by (K.y(E) pe. (e).
In addition to the verification problem, IBP can be used to perform certified training such that the specification is likely to be satisfied in the first place at the end of training. For this, the overall training loss would consist of two terms: usual classification loss like cross-entropy that maximizes log likelihood 1(zK, Awse), and loss term that encourages satisfiying the specification 1(5x (c). fuue)) Here, (c) = k (c) (if for certified training can be represented as:
+ Vlue) and & (c) = KK (c) (if Vise). So, the overall loss
== xl(= /mm) + (1 - ()x((x(()) lime))
As reported in (Gowal et al., 2018; Zhang et al., 2019b; Xu et al., 2020), to stabilize training, x is varied during training such that for few initial epochs x 1 (normal training) and its value is slowly increased so that robust training kicks in gradually.
A.2 PROOF OF THE THEOREM
Theorem 1 (Verification. The verifier can verify whether the trained model from the designer satisfies the specifications, by generating a proof of the same). Let x E X be the input, 2g = e(x) be the latens code, be a feed-forward neural network with bounded derivatives, and St,in {z: || =0,1lp S e} be the set of latent perturbations. Then, the verifier is guaranteed to be able to generate a proof for verifying whether the linear specification on output logits cº =g + d 0 Vz S =K fa(z) (corresponding to unit test i) is satisfied.
Proof. Revisiting the notation in section 2, consider a network mapping f f: + 2 - (denote by Sa : >> y the downstream task) and generative model mapping 9 - 2 - &, such that f and g share common latent space Z by virtue of having common encoder e(-) (Figure 2). Here we provide a proof based on fully-connected feed-forwand forward network architecture for fa. an arbitrary architecture for If and using the interval-Bound Propagation (IBP) method for obtaining output bounds. For a general network architecture for fa, we refer the reader to a recent library AutoLirpa (Xu al., 2020).
Given input r E X, let 2g = e(x)) be the latent code, and S,,i = f2 ||2 2n,1lp S c) be the set of latent perturbations. Let, be dddiimesiional. Here, z, denotes the dimensions(s) of z that
15

---

perturbed, such that 2d-i 24. Without loss of generality, 2 to be contiguous. The verifier can generate proof by searching for a worst-case violation
assume all the dimensions in
c²z² + s.t. 2 = hu k= + by) k 00 K =ESL,
max
For the sake of simplicity, first consider the case where j = d i.c. all dimensions are perturbed. So, Sin {=9 || zBllp I Here, we have denoted by =0 the latent code in 2 space, and will h =W + by) k = 0, .... K - 1 to denote the subsequent intermediate variables in h : Z + y. Based on Holder's inequality we can obtain linear bounds for the first variable z!,
use 2
z= as follows:
z -ell + W0 +b;; = 11, W +b; 1/p 1/q=1 =
Here, 11 denotes computation of the norm for each row in the matrix and the values of W w can be computed with the functions described in Appendix A.1 of (Xu et al., 2020) depending on the type of non-linearity in h (-).
Now, when j * d i.. only j dimensions are perturbed, without loss of generality, we can consider all of the dimensions to be contiguous such that z" zd-s 2/. We can then compute [z), x/] as follows:
Σ =
19 + W + D9: zl ='= WZ= +b9; 1/p+ 1/9
We can compute z follows:
= 11 No 2d-1 BD)
Finally, obtain [=],²'] as follows:
= 2
z 2/- zz
Now, given the bound [x] 21| and the relation *+1 1 = t4 (W + bb) 0, K 1, we can
obtain the bound for K zk] by recursively computing successive bounds as follows:
z
+
x+
W
+ b
W
z -
z x
zh+1
W
+ b
W
2
Here we have assumed that hy is an element-wiss wise monotonic function, which is is for for most common actions like ReLU, tanh, sigmoid etc. Using [2 zx1, the verifier can construct an upper
and lower bound on the solution of m. cT 22 + d
T K <I<EK
max
+
important to note that the proof the verifier generates to check whether the specification holds relies on computing an upper bound æ and checking whether the upper bound o < 0. If the upper bound is not tight, then there may be cases when the true solution Theorem 1 is indeed satisfied) while the upper bound , > 0 (implying the verifier will state that the specification is not satisfied). This form err on the side of caution is a natural consequence of proof by generating worst example, and follows directly from IBP.
(meaning the specification in
16

---

A.3 IMPLEMENTATION DETAILS
We use a 32-GB V100 GPU for all our experiments. The implementation in Python with combination of 1ePPyToe Torch and Tensorflow deep learning libraries. Details about the specific experiments are mentioned below:
ImageNet. We use a pretrained BigBiGAN https://colate https///colabb.eeeacch....ggg. httpaiAAAtttssroooob.eeeeeerrrae google. com/ github/Eensor's hwtp///ww///////tto hhub///lo/mastteee
hub. .ipynb as the base generative model, such that the encoder encodes images the latent space. The downstream classifier from the latent space consists of 5 fully connected layers with ReLU non-linearities. We use ADAM optimizer with learning rate of 0.0005. For each value of T2 perturbation in Table 1, we train for 100 epochs, using the AutoLiRPA library's implementation of CrownIBP nttps:// /ttp:://iitt hub. com/ Kaid diiii aut o LiRPA. The first 5 epochs are standard training for warming up, and the rest of the epochs are robust training. The training time for 100 epochs is about 8 hours.
CheXpert. We train PGGAN https: / com/ con//haarraasprogrssssiii ca_gann
growing of gans on the CheXpert dataset. In order to learn latent embeddings from raw images, we need an inference mechanism. For this, we perform GAN inversion of this pretrained generator, using https: g/tttbb...b......//g com/ dav dbau/ gansee wnweeeeennngnne mggblll blcb/ release/ run_ invert sh to obtain encoder for projecting images to the latent space. The downstream classifier from the latent space consists of fully connected layers with ReLU non-linearities. We use ADAM optimizer with a learning rate of 0.0005. For each value of L2 perturbation in Table 1, we train for 100 epochs, using the AutoLiRPA library's ' implementation of CrownIBP https: : / / thub. com/ hae//aiica miiiiiuuaaat autoo LiRPA. The first 5 epochs standard training for warming up, and the rest of the epochs are robust training. The training time for 100 epochs is about 7 hours.
LSUN and FFHQ. We train StyleGAN models https:/ /ttp://gttthas. com NVVabbs/ stylegan2 ada pytorch on the FFHQ and LSUN datasets, and perform GAN inver- sion using https:://itte
to obtain respective encoders disentangled latent space. The downstream classifier from the latent space consists of 5 fully connected layers with ReLU mon-linearities We use ADAM optimizer with learning rate of 0.0005. For each value of e L22 perturbation in Table 1, we train for 100 epochs, using the Auto library's implementation of CrownIBP https: / //g thub. com / Ka d Xu /u/auto, auto_IiRPA The first 5 epochs are standard training for warming up, and the rest of the epochs are robust training. The training time for 100 epochs is about 7 hours each.
com hwam///enorre///ee/
real
0.5
1.0
2.0
4.0
6.0
Figure 5: Each column shows images from aprticular category that are presented in the HITs for the Amazon MTurk Experiment. For each HIT, we show pairs of images such that one image is from the real category and the other image is genertaed image from one of the other categories.
17

---

Table 6: In this experiment, we pre-train the classifier through certified training with < = 1.0 and perform verification for different values of e. Verified Error fraction on the FFHQ and LSUN datasets. For FFHQ. the task is classify whether eyeglasses are present, and the unit tests correspond to variations with respect pose and expression of the face. For I.SUN Tower, the task is classify whether green vegetation is present the tower, and the unit tests correspond to variations with respect to clouds the sky and brightness the scene. Results are on the held-out test set of the dataset and correspond to training the classifier with the same during training. S.D. is over four random seeds of training. Lower is better.
FFHQ
0.5
(50%5 = 1.5%) - 2.0(100%) = 3.0(150%)
Pose test
0.005=0.01
0.2011005
0.39=0.02
0.51=0.004
0.56=0.05
Expression test
0.005++0.005
0.13++0.04
0.33++0.04
0.57++0.01
0.58=0.03
LSUN Tower 6 = 0.5(25%)
6 1.0(50%)
= 1.5(75%)
= 2.0(100%)
== 3.0(150%)
Cloud test
0.01+00.01
0.14+00.02
0.36+10.04
0.47++0.06
0.52=0.04
Brightness test
0.005=10.01
0.18:10.03
0.38+10.55
0.52:10.06
0.57=0.04
A.4 DETAILS ABOUT THE AMAZON MTURK STUDY
Here we provide more details about the setup for the Amazon MTurk study whose results we presented in Fig. 4. We designed each Human Intelligence Task (HIT) to be comprised of 10 pairs of images sequentially presented, such that each pair consists of real and generated image. For different values of we show examples of images from each of the categories presented in the HITs through Fig. 5.
We recruited participants only from the "Master Worker" category. As defined in https: /www. mturk. com/ htt///woww//eeeeeeeee /hepp. "a Master Worker is top Worker of the MTurk marketplace that has been granted the Mechanical Turk Masters Qualification." " These Workers have consistently demonstrated high degree of success in performing wide range of HITs across large number of Requesters. The participants were paid 0.5 dollars for each HIT (i.e. for 10 pairs of images). Each pair was shown to the participants for 2 seconds, after which the images disappeared from the screen, but they could take as much time as they wanted to decide and form a response. We restricted the number HITs per participant to 20.
The participants were shown the exact text below: Choose the more realistic image between options A and B. For each pair, you have 5 seconds view the image and unlimited time make the decision. Each task consists of 10 image pairs.
A.5 RESULTS OF VERIFICATION WITH PRE-TRAINED CLASSIFIER
In this experiment, we pre-train the classifier through certified training with c = 1.0 and perform verification for different values e.
A.6 EXTENDED RELATED WORKS
Adversarial robustness. Multiple prior works have studied a subset of the audit problem in terms of adversarial robustness (Carlini et al., 2019; Jia et al., 2019; Gowal et al., 2018; Zhang et al., 2018; Singh et al., 2019; Weng et al., 2018; Singh et al., 2018; Wang et al., 2018; Raghunathan et al., 2018a;b; Dvijotham et al.). Adversarial examples are small perturbations the input (for images the level of pixels; for text at the level of words) which mislead the deep learning model. Generating adversarial examples has been widely studied (Zhao et al., 2017: Athalye al., 2018; Carlini & Wagner, 2017a;b: Goodfellow et al., 2015; Madry et al., 2018; Papernot et al., 2016; Xiao et al., 2019b: 2018b:e; Eykholt et al., 2018; Chen al., 2018; Xu et al., 2018: Zhang et al., 2019a) and recent approaches have devised defenses against adversarial examples (Guo et al., 2018: Song et al., 2017; Buckman al., 2018; Ma al., 2018: Samangouei et al., 2018; Xiao et al., 2018 8a; 2019a; Xie et al., 2020; Cohen et al., 2019; Gowal et al., 2018; Zhang et al., 2019b). Some of these approaches like (Gowal et al., 2018) and (Zhang et al., 2019b) provide provable verification bounds on an output specification based on the magnitude of perturbations in the input. These approaches have largely showed robustness and verification to norm bounded adversarial perturbations without trying to verify with respect to more semantically aligned properties of the input. Some papers (Mohapatra et al., 2020; Ruoss et al., 2020; Balunović et al., 2019) consider semantic perturbations like rotation, translation, occlusion, brightness change etc. directly in the pixel space, the range of semantic
18

---

variations that can be considered are more limited. This is distinction with AuditAI, where by perturbing latent codes directly (as opposed to pixels), the range of semantic variations captured much larger. In (Wong & Kolter, 2020), the authors model perturbation sets for images with perturbations (like adversarial corruptions) explicitly such that a conditional VAE can be trained. While, for Audi tAT, we identify directions of latent variations that present in pre-trained GANs, and as such do not require explicit supervision for training the generative model, and can scale to datasets like ImageNet and CheXpert, and high dimensional images like that in FFHQ.
Neural network interpretability. With the rapid rise of highly reliable generative models, many recent works have sought to analyze the latent space of such models and how the latent space can be manipulated to obtain semantic variations in the ouiput image. (Bau et al., 2021) use text embeddings from CLIIP (Radford et al., 2021) to perform directed semantic changes in the generated images, for example introduction of a particular artifact in the room change in the color of a bird. (Bau et al., 2020a; Shen & Zhou, 2020; Shen et al., 2020) identify latent code dimensions corresponding to pose of the body/face, expression of the face, orientation of beds/cars in the latent space of generative models like PGGAN (Karras et al., 2017) and StyleGAN (Karras et al., 2019). By manipulating these latent dimensions, the output images papers that sought to understand what the neural network weights are learning in the first place (Bau al., 2020b; Yeh et al., 2020) in an attempt to explain the predictions of the networks.
be controllably varied. In addition, there have been other
Auditing, While we proposed a framework for auditing of DI models, a complimentary problem is to audit datasets that used for training DL models. In (Gebru et al., 2018), the authors proposed checklist of questions to be answered while releasing datasets that relate to the motivation, composition, collection process, processing, and distribution of the dataset. In (Mitchell et al., 2019), the authors proposed a checklist of questions that should be answered while releasing DL models that relate to their motivation, intended use cases, training details, and ethical considerations. These papers formed part of our motivation and provided guidance in formalizing the audit problem for DL models.
19


Reviews:
[{"review_id": "ad5EvY11L28", "reviewer": "Reviewer_rsGL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tackles an important problem and provides a comprehensive discussion and evaluation of the approach. Certified training and verification is quite important for developing reliable models going forward. I detail key strengths, weaknesses, and additional questions below.\n\n\n### Strengths\n- Extending IBP to the latent code is straightforward and simple. \n- The breadth/scope that this work tackles is impressive. The paper shows how to verify a particular behavior like brightness, etc, and also how to train in a certified manner. \n- The scope of the empirical assessment in the work is also impressive. The paper tests on domain shift, ImageNet, Chest Xrays, and FFHQ. In each case, demonstrating convincing performance of the modified IBP.\n\n\n### Weaknesses\n- Toy setting with known ground truth. There is a simple experiment that the authors can run to provide a sanity check that their variation of IBP 'works'. Why not train a toy model that specifically violates a unit test of interest and show that the entire scheme here recovers the violation? \n- The principal assumption in this work is that the latent code and the corresponding generator can decompose the input into semantically meaningful concept. Even further, the assumption is that each dimension of the latent code only encodes for a single concept. Contemporary evidence shows that this assumption might not exactly be true. While this issue doesn't invalidate this work, I think the authors to make this assumption/requirement more explicit.\n- User study. The study is interesting, but I am not sure what to take away from this part of the work. First, I don't think Turkers are quite equipped to judge Chest XRays. However, even if we assume that they are, showing indistinguishability between real and generated images does not demonstrate the effectiveness of AuditAI.\n- I am confused that Theorem 1 does not have assumptions on $f$. It seems like what IBP does (not an expert on this technique) is to use a bound on the logits to bound the output of the network. Wouldn't this require assumptions on the activations or some form of bound on the Lipschitz constant of the function? \n- It is unclear to me how the search procedure in IBP actually works in practice. Can this information be added to the appendix?\n- What does, \"the corresponding convex bound of the variations of $\\mathcal{S}^{0}_{i, in}$ in image space\" mean?", "summary_of_the_paper": "This paper presents, AuditAI, a system for verifying that a trained model meets a particular `specification', and also for certified training. The system uses a generative model to construct embeddings for inputs and then tests whether these embedding vectors belong to a particular set. Membership in the pre-specified set is reduced to satisfying the test. The key technical insight in this work is applying the interval bound propagation technique to embedding/latent codes in order to verify semantic behaviors of trained models.", "summary_of_the_review": "I lean towards an acceptance for this work since it provides a convincing demonstration of IBP for latent embeddings, which means more interesting properties beyond adversarial robustness on the input space can be verified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YASVgQ-QM3q", "reviewer": "Reviewer_Qgvd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The paper is written clearly.\n\nWeakness:\n- The paper seems to be a natural extension of IBP, Advmix [1]\n- I am not sure whether the distinction with [2] is clear (in appendix A.6). I think modeling perturbation sets for images with perturbations (like adversarial corruptions) explicitly can capture the specification better. \n- For the perturbation in the semantic space of a generative model, it is not clear to me whether this is a semantic preserving perturbation. Also it is not clear whether the latent for a particular GAN model can capture all the variations. Usually, the diversity of the GAN is quite limited. \n- I don’t find the unit test arguments to be very novel as it is a standard verification protocol. \n- The name AuditAI is not formal. \n- Minor: for few references, the use of citep, citet is mixed.\n- Minor: theorem 1 “(“ should be removed.\n\n[1] https://arxiv.org/abs/1912.03192\n\n[2] https://openreview.net/pdf?id=MIDckA56aD\n", "summary_of_the_paper": "This paper presents AuditAI that can do certified training, verification under semantic specifications based on a generative model. \nThe authors extended IBP to latent space variations and evaluate on a wide range of datasets. \n", "summary_of_the_review": "Overall, the novelty of the paper is a bit limited and I am not sure whether the perturbation on the latent space of generative models indeed captures all the possible semantic specifications. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "U0991wlmLmK", "reviewer": "Reviewer_1Tmi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Major Issues\n* At times the formal foundation is not precisely specified so a detailed\n   understanding is at times difficult, e.g.:\n    * Page 4: $L = L_{task} + ɣ L_{spec} + δ L_{gen}$ -> gamma and delta not\n      explained\n    * Page 4: Variables used in paragraph \"Task Performance\" are only introduced\n      in the next paragraph \"Verifiablity\" (e.g. $z_K$, $y_{true}$)\n* Page 5: Not clear how the paragraph \"Interpretability\" which discusses\n   $L_{gen} is related to interpretability.\n* Page 8; Section 4.5: Authors are talking about \"certified images\"; so far the\n   certification was about the training not individual images, the intend of\n   this section is not clear; also is not a robustness against adversarial\n   attacks also a property which can be seen as a variation which is the target\n   of the framework.\n* Page 8; Section 5: The authors perform a human study to evaluate whether the\n   generated images are realistic for the human eye. However I am unsure whether\n   this is a good metric for evaluation since realistic images for the human eye\n   are known to not be equivalent to useful images for machine learning datasets\n   (c.f.g. adversarial attacks). Also the participants of the study are of\n   unknown expertise and the chosen dataset was that of Chest-X-ray images.\n\n## Discussion\nThe paper contributes to a solution of a problem that is both relevant and in\nscope of the conference. The authors motivate the paper and clearly state the\ncontributions as well as the followed approach. While some rework of the formal\nfoundation (see `Points Against: 3.) would be good to increase understandability\nthey are sound and allow a deep understanding of the approach and also convince\nthe reader that the approach is well thought out. However the usage of a\ngenerative model to create the inputs to check boundaries does limit the\nconfidence in the framework since the reliability of the generative network is\nnow at question. Which means the reliability of the framework depends again on\nanother machine learning component. The evaluation of this component, the\ngenerative network, was done by performing a human study about how realistic the\nimages are for the human eye and was also performed by participants of unknown\nexpertise. The specification of the encoding function $e()$ is not clear aside\nfrom the used example of the change in degree and should be discussed further,\nas this is also an important aspect in the framework. Without clear\nspecification of $e()$ and assurance that this in fact represents the variation\nwhich needs to be audited the result of audit is questionable. Non-the-less the\ncontribution is worth publishing as this represents a sound foundation for\nfuture research.\n\n## Minor issues\n* Figure 1: The mapping of the three parts of the figure to the explanation\n  could be improved.\n* No structural overview of the paper at the end of the introduction.\n* Page 2; last paragraph: \"… 25%larger …\" -> space missing\n* across the whole paper: \"upto\" -> up to\n* Page 5; first paragraph: \"tigher\" -> tighter\n* Page 7; Table 4 is referenced before Table 3 in the text, so these two tables\n  should be swapped in the tex document.\n* Page 8 and 9: \"AuditAI\" is not in monospace font\n* the authors at times claim to certify, however this is in my opinion to\n  strong, most of the time they speak of auditing which is the better\n  description", "summary_of_the_paper": "This paper presents a framework for auditing deep learning models in regards to\na specification. The goal is to increase the confidence in a trained model. The\nauditing is performed by generating variations in input data to evaluate the\nboundaries in the input for an expected output. An evaluation was performed\nagainst a pixel-based training approach and under consideration of 4 different\ndatasets.", "summary_of_the_review": "## Points in Favor\n+ Relevant problem and in scope of the conference.\n+ Presentation of a framework for auditing deep learning models in respect to\n   given specification.\n+ Formally sound foundation of the framework and its mathematical workings.\n+ Generally the introduction of the framework is given in stages with\n   relaxing assumptions, so the understandability is good.\n+ Experiments show the improvement with the framework.\n\n## Points Against\n- Generative Network used to create the variations: while this strengthens the\n   confidence in the model it does move the reliability of the framework to\n   the GAN as well.\n- The framework is dependent on the encoding function e() and it is not clear\n   whether this is always specifiable.\n- Partly, the formal foundation is not precisely specified", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "N6zPc8eeAXm", "reviewer": "Reviewer_PLLS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n1. This paper proposes a new direction AI model audit. I think this is very useful direction, in particular, for industry. Rejecting making a decision (e.g. reject making decision for an image of bad quliaty and ask the user to re-submit a better image) is much better than making a wrong decision. \n2. The proposed solution is convincing for me and the experiments show the efficacy of the solution.\n\nCons\n1. The classification model and the generative model share the same encoder. You cannot directly apply the proposed method to any exsiting network. For example, the classification backbone is ResNet-50. You cannot connect an encoder (for disentanglement learning) to this ResNet-50, or first few layers of ResNet50 works as the encoder.\n2. The key theoritical barries are addressed by an off-the-shelf method, IBP.\n3. The method cannot be applied to other tasks, e.g. object detection, segmentation, etc. At least, no supportive experiments for other tasks.\n\nQuestions and Suggestions\n1. Can you translate the face rotation angels are within 30 degree and maintain 95% accuracy to the representation with  F(x,y) <= 0 and S_{i, in}.\n2. In practice, many people use the uncertainty/quality/confidence score to do AI model audit. I would suggest to compare with them in Related Work section.", "summary_of_the_paper": "This paper proposes a new and useful direction: AI model audit. The proposed solution (framework) is very interesting. The experiments show the effectiveness of the proposed method.", "summary_of_the_review": "Despite my concerns, I think this is a VERY valuable work for the society. The paper is well written and organized.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper discusses the auditing of AI systems which is clearly a very important topic. The approach targets mainly image classification systems where the main idea is to do the certification not in pixel space but in a latent space of a generative model. For the certification the authors use the existing CROWN-IBP for cubes around test points in latent space. The authors report results on several datasets using existing methods for the generative models.

Most of the reviewers liked this work even though all of them also raised quite similar concerns. The positive reviews were to some extent based on the fact that auditing of AI systems is a very important problem - and I completely agree on this. On the other hand auditing of AI systems, in particular of safety critical systems as discussed with the chest x-ray application in this paper, is a serious problem where a paper has to fulfill very high standards. In particular, being transparent about its limitations is crucial which in my opinion does not hold for the current version of this paper.

My main conerns are:
- the motivating example (varying the angle from which a face is seen by less than 30 degress) is misleading as such a precise threat model is not feasible with the generative models discussed later on (see last bullet point)
- there is no discussion on the potential trade-off regarding test accuracy training the classifier i) in latent space, ii) with CROWN-IBP using a rather shallow model. This is important to judge the utility of this approach - it has to be transparent what the trade-off is regarding test accuracy and what kind of price one has to pay in terms of test accuracy for the certificates.
- the deployment check discussed in Section 2.4 is nowhere implemented as later on only boxes in latent space around individual test points are checked but not a fixed box
- the main problem connected to what I mentioned above with the motivating example is that the latent dimensions have to be interpreted by humans. I doubt that this can be made fully transparent to humans what such a certificate in latent space meand and thus to give very precise labels to the latent dimensions like "brightness" is in my point of view misleading. It suggests a much more finegrained semantic control than what is available and to use such inprecise and potentially wrong ``labels'' is highly problematic for an auditing process as e.g. this paper does not contain any guarantee against "brightness"-changes. For safety critical systems this is an important distinction.

As a minor point there is little technical novelty (certification with CROWN-IBP is done with an existing library) but this plays no role in the decision.

In total the paper is not ready yet for publication. The authors should revise their manuscript by addressing the points above, being fully transparent about what the paper can do and what it cannot do. I suggest to do less examples but provide more details on how the authors envision the auditing/certification process for the given application, in particular also with the discussion about rejection of inputs not lying in certified areas (which has not been done in the present paper).
Decision: Reject