Auditing AI models for Verified Deployment under Semantic Specifications

1 INTRODUCTION . Deep learning ( DL ) models are now ubiquitously deployed in a number of real-world applications , many of which are safety critical such as autonomous driving and healthcare ( Kendall et al. , 2019 ; Miotto et al. , 2018 ; Senior et al. , 2020 ) . As these models are prone to failure , especially under domain shifts , it is important to know when and how they are likely to fail before their deployment , a process we refer to as auditing . Inspired by the failure-mode and effects analysis ( FMEA ) for control systems and software systems ( Teng & Ho , 1996 ) , we propose to audit DL models through a sequence of semantically-aligned unit tests , where each unit test verifies whether a pre-defined specification ( e.g. , accuracy over 95 % ) is satisfied with respect to controlled and semantically meaningful variations in the input space ( e.g. , the angle relative to the camera for a face image ) . Being semantically-aligned is critical for these unit tests to be useful for the auditor of the system to plan the model ’ s deployment . The main challenge for auditing DL models through semantically-aligned unit tests is that the current large-scale DL models mostly lack an interpretable structure . This makes it difficult to quantify how the output varies given controlled semantically-aligned input variations . While there are works that aim to bring interpretable formal verification to DL models ( Henriksen & Lomuscio , 2020 ; Liu et al. , 2019 ) , the scale is still far from the millions if not billions of parameters used in contemporary models ( He et al. , 2016 ; Iandola et al. , 2014 ; Brown et al. , 2020 ) . On the other hand , auditing has taken the form of verified adversarial robustness for DL models ( Samangouei et al. , 2018 ; Xiao et al. , 2018a ; Cohen et al. , 2019 ) . However , this has mostly focused on adversarial perturbations in the pixel space , for example , through Interval Bound Propagation ( IBP ) , where the output is guaranteed to be invariant to input pixel perturbations with respect to Lp norm ( Gowal et al. , 2018 ; Zhang et al. , 2019b ) . While these approaches are much more scalable to modern DL architectures , the pixel space variations are not semantically-aligned , meaning they do not directly relate to semantic changes in the image , unlike in formal verification . Consider a unit test that verifies against the angle relative to the camera for a face image . A small variation in the angle ( e.g. , facing directly at the camera versus 5◦ to the left ) can induce a large variation in the pixel space . Current certified training methods are far from being able to provide guarantees with respect to such large variations in the pixel space with reasonable accuracy . Our Approach . In order to overcome the above limitations , we develop a framework for auditing , AuditAI . We consider a typical machine learning production pipeline ( Fig . 1 ) with three stages , the design and training of the model , its verification , and finally deployment . The verification is crucial in determining whether the model satisfies the necessary specifications before deployment . We address the gap between scalability and interpretability by proposing to verify specifications for variations directly in a semantically-aligned latent space of a generative model . For example in Fig . 1 , unit test 1 verifies whether a given face classification model maintains over 95 % accuracy when the face angle is within d◦ , while unit test 2 checks under what lighting condition the model has over 86 % accuracy . Once the verification is done , the auditor can then use the verified specification to determine whether to use the trained DL model during deployment . For semantically-aligned latent variations , we create a bridge between the generative model and the DL model such that they share the same latent space . We incorporate a variant of IBP ( Zhang et al. , 2019b ) for verification with respect to perturbations in the latent space . Further this leads to a tighter and a much more practical bound in the output space compared to pixel-based certified training . We also show that AuditAI can verify whether a unit test is satisfied by generating a proof for verification based on bound propagation . Fig . 1 gives an overview of our auditing framework and Fig . 2 elaborates on the generative model bridge . Summary of Contributions : . 1 . We develop a framework , AuditAI for auditing deep learning models by creating a bridge with a generative model such that they share the same semantically-aligned latent space . 2 . We propose unit tests as a semantically-aligned way to quantify specifications that can be audited . 3 . We show how IBP can be applied to latent space variations , to provide certifications of semanticallyaligned specifications . We show that AuditAI is applicable to training , verification , and deployment across diverse datasets : ImageNet ( Deng et al. , 2009 ) , Chest X-Rays ( Irvin et al. , 2019 ; Wang et al. , 2017 ) , LSUN ( Yu et al. , 2015 ) , and Flicker Faces HQ ( FFHQ ) ( Karras et al. , 2019 ) . For ImageNet , we show that AuditAI can train verifiably robust models which can tolerate 25 % larger pixel-space variations compared to pixel-based certified-training counterparts for the same overall verified accuracy of 88 % . The variations are measured as L2 distances in the pixel-space . The respective % increase in pixelspace variations that can be certified for Chest X-Rays , LSUN , and FFHQ are 22 % , 19 % , 24 % . We conclude with a human-study of the quality of the generative model for different ranges of latent-space variations revealing that pixel-space variations up to 62 % of the nominal values result in realistic generated images indistinguishable from real images by humans . 2 AUDITAI : A DEEP LEARNING AUDIT FRAMEWORK In this section , we describe the details of our framework , AuditAI outlined in Fig . 1 for verifying and testing deep models before deployment . We propose to use unit tests to verify variations of interest that are semantically-aligned . The advantage of AuditAI is that the verified input ranges are given by several semantically aligned specifications for the end-user to follow during deployment . In the following sub-sections , we formally define unit tests , outline the verification approach , and describe the specifics of AuditAI with a GAN-bridge shown in Fig . 2 . 2.1 UNIT TEST DEFINITION . Consider a machine learning model f : X → Y that predicts outputs y ∈ Y from inputs x ∈ X in dataset D. Each of our unit test can be formulated as providing guarantees such that : F ( x , y ) ≤ 0 ∀x s.t . , ei ( x ) ∈ Si , in , y = f ( x ) ( 1 ) Here , i subscripts an unit test , encoder ei extracts the variation of interest from the input x , and Si , in denotes the set of range of variation that this unit test verifies . If the condition is satisfied , then our unit test specifies that the output of the model f ( x ) would satisfy the constraint given by F ( · , · ) ≤ 0 . For example , x could be an image of a human face , and f could be a classifier for whether the person is wearing eyeglasses ( f ≤ 0 means wearing eyeglasses and f > 0 means not wearing eyeglasses ) . Then ei ( · ) could be extracting the angle of the face from the image , and Si , in could be the set { ei ( x ) | ∀x |ei ( x ) | < 30◦ } constraining the rotation to be smaller than 30◦ . And F ( x , y ) = −f ( x ) f ( x0◦ ) ≤ 0 says that our classifier output would not be changed from the corresponding output of x0◦ , the face image of x without any rotation . In this case , when the end-user is going to deploy f on a face image , they can first apply ei ( · ) to see if the face angle lies within Si , in to decide whether to use the model f . 2.2 VERIFICATION OUTLINE . Given a specification F ( x , y ) ≤ 0 , we need to next answer the following questions : 1 . How do we obtain the corresponding components ( ei , Si , in ) in Eq . ( 1 ) ? 2 . What can we do to guarantee that F ( x , y ) ≤ 0 is indeed satisfied in this case ? For the sake of illustration , we first consider a scenario with a less realistic assumption . We will then discuss how we can relax this assumption . In this scenario , we assume that we are already given the encoder ei and a corresponding generator gi that inverts ei . Continuing the previous example , ei can extract the face angle of a given face image . On the other hand , gi ( x0◦ , d◦ ) would be able to synthesize arbitrary face image xd◦ that is the same as x0◦ except being rotated by d◦ . Given the generator gi and the encoder ei , we propose to obtain Si , in building on interval bound propagation ( IBP ) ( Gowal et al. , 2018 ) . We include a treatment of IBP preliminaries in Appendix A.1 . Here , Si , in is the largest set of variations such that the specification F ( x , y ) ≤ 0 is satisfied . Given a set of variations Si , in , IBP-based methods can be used to obtain a bound on the output of the network Si , out = { y : ly ≤ y ≤ uy } and it can be checked whether the specification F ( x , y ) ≤ 0 is satisfied for all values in Si , out . We can start with a initial set S0i , in and apply gi to this set . Then , we can first find out what would be the corresponding convex bound of variations of S0i , in in the image space X . We can subsequently propagate this bound in X through f to Y to get S0i , out and check whether F ( x , y ) ≤ 0 is satisfied . By iteratively searching for the largest set Sji , in such that F ( x , y ) ≤ 0 is still satisfied by S j i , out , we can obtain Si , in given F . 2.3 CERTIFIED TRAINING THROUGH LATENT REPRESENTATION . In the previous section , we are able to find the set Si , in such that Eq . ( 1 ) is satisfied given specification F , encoder ei , generator gi . However , there are several challenges that limit the practical application of this scenario . First and foremost , while significant progress has been made in generative models , especially controlled generation ( Brock et al. , 2018 ; Karras et al. , 2017 ) , the assumption of having gi apriori is still unrealistic . In addition , since the propagated bound is convex , one can imagine that the bound in X ( high dimensional image-space ) would be much larger than needed . This leads to a much narrower estimate of Si , in . While it could be true that Eq . ( 1 ) is satisfied , we still need a non-trivial size of Si , in for the framework to be useful . For example , if Si , in constrains face angle rotations to be within 1◦ then it may not be practical for the end-user . For auditing ML models thoroughly , we require clear separation between the development and testing phases ( i.e . black-box verification ) . However , it might also be desirable in practice to train the model in such a way that the unit tests are likely to be satisfied in the first place . In this case , we do have a tighter connection between the designer and the verifier . We show that by relaxing the black-box constraint , we can simultaneously address the previously listed challenges in a white-box setting . Now that we have access to the internal working and design of the ML model f , we propose to bypass the image generation step in our verification process through the use of latent disentangled representations . More specifically , consider a ML model mapping f : X → Z → Y ( denote by fd : Z → Y the downstream task ) and a generative model mapping g : X → Z → X̂ , such that f and g share a common latent space Z by virtue of having a common encoder e ( · ) ( Fig . 2 ) . Since Si , in is a subset of the range of e , we have Si , in ⊆ Z . This further implies that we do not have to apply IBP through g and the whole f . Instead we only have to apply it through fd , which does not involve the expansion to the pixel space X . We show in the experiment ( Section 4 ) that this is important to have a practically useful Si , in . In addition , AuditAI also alleviates the need of having a perfect generative model , as long as we can learn a latent space Z through an inference mechanism . Learning the latent space . There are three requirements for the latent space . First , it should be semantically-aligned for us to specify unit tests in this space . A good example is disentangled representation ( Higgins et al. , 2018 ; Tran et al. , 2017 ) , where a set of dimensions of the latent code would correspond to a semantically aligned variation of the original image . As shown in Figure 2 , a latent dimension could correspond to the pose variation , and we can select ei to be the value of this dimension , and Si , in as the proper range that the model would be verified for . Second , our model should be verifiable , where the latent space is learned such that the corresponding Si , in is as large as possible given a specification function F . In this case , AuditAI can be seen as doing certified training ( Zhang et al. , 2019b ) to improve the verifiability of the model . Finally , the latent space should be able to perform well on the downstream task through fd . Combining these three criteria , our full training criteria : L = Ltask + γLspec + δLgen combines three losses Lgen , Lspec , and Ltask that would encourage interpretability through the latent space of a generative model , verifiability , and task performance respectively . γ and δ are relative weights ( between 0 and 1 ) for the loss function terms . We explain each of the losses below : Verifiability ( Lspec ) . Let fd be a feedforward model ( can have fully connected / convolutional / residual layers ) with K layers zk+1 = hk ( Wkzk + bk ) , k = 0 , ... , K − 1 , where , z0 = e ( x ) , zK denotes the output logits , and hk is the activation function for the kth layer . The set of variations Si , in could be such that the lp norm for the ith latent dimension is bounded by ϵ , i.e . Si , in = { z : ||zi − z0 , i||p ≤ ϵ } . We can bound the output of the network Sout = { zK : lK ≤ zK ≤ uK } through a variant of interval bound propagation ( IBP ) . Let the specification be F ( z , y ) = cT zK + d ≤ 0 , ∀z ∈ Si , in , zK = fd ( z ) . For a classification task , this specification implies that the output logits zK should satisfy a linear relationship for each latent space variation such that the classification outcome argmaxi zK , i remains equal to the true outcome ytrue corresponding to z0 . To verify the specification , IBP based methods search for a counter-example , which in our framework amounts to solving the following optimization problem , and checking whether the optimal value is ≤ 0. max z∈Sin F ( z , y ) = cT zK + d s.t . zk+1 = hk ( Wkzk + bk ) k = 0 , ... , K − 1 ( 2 ) By following the IBP equations for bound-propagation , we can obtain upper and lower bounds [ zK , zK ] which can be used to compute the worst case logit difference zK , y − zK , ytrue between the true class ytrue and any other class y . We define ẑK = zK , y ( if y ̸= ytrue ) ; ẑK = zK , ytrue ( otherwise ) . Then , we can define Lspec as Lspec = CE ( ẑK , ytrue ) . In practice , we do not use IBP , but an improved approach CROWN-IBP ( Zhang et al. , 2019b ) that provides tighter bounds and more stable training , based on the AutoLiRPA library ( Xu et al. , 2020 ) . We mention details about this in the Appendix . Task Performance ( Ltask ) . Let , CE ( · ) denote the standard cross-entropy loss . Ltask captures the overall accuracy of the downstream task and can be expressed as Ltask = CE ( zK , ytrue ) . Generative Model ( Lgen ) . Finally , we have a loss term for the generative model , which could be a VAE , a GAN , or variants of these . If the model is a GAN , then we would have an optimization objective for GAN inversion . An assumption is being able to learn a disentangled latent space with semantically aligned concepts . Let the overall loss be Lgen which could also encapsulate other losses for more semantically-aligned disentanglement . Typically we would train this generative model on a large amount of unlabelled data that is not available for training the classifier we are auditing .

Summary: This paper proposes a new and useful direction: AI model audit. The proposed solution (framework) is very interesting. The experiments show the effectiveness of the proposed method.