Abstract: I. J. Good's intelligence explosion theory predicts that ultraintelligent agents will undergo a process of repeated self-improvement; in the wake of such an event, how well our values are fulfilled would depend on the goals of these ultraintelligent agents. With this motivation, we examine ultraintelligent reinforcement learning agents. Reinforcement learning can only be used in the real world to define agents whose goal is to maximize expected rewards, and since this goal does not match with human goals, AGIs based on reinforcement learning will often work at cross-purposes to us. To solve this problem, we define value learners, agents that can be designed to learn and maximize any initially unknown utility function so long as we provide them with an idea of what constitutes evidence about that utility function.

Learning What to Value

Agents and Implementations Traditional agents  [2] [3] interact with their environments cyclically: in cycle k, an agent acts with action y k , then perceives observation x k . The interaction history of an agent with lifespan m is a string y 1 x 1 y 2 x 2 ...y m x m , also written yx 1:m or yx ≤m . Beyond these interactions, a traditional agent is isolated from its environment, so an agent can be formalized as an agent function from an interaction history yx <k to an action y k . Since we are concerned not with agents in the abstract, but with very powerful agents in the real world, we introduce the concept of an agent implementation. An agent implementation is a physical structure that, in the absence of interference from its environment, implements an agent function. In cycle k, an unaltered agent implementation executes its agent function on its recalled interaction history yx <k , sends the resulting y k into the environment as output, then receives and records an observation x k . An agent implementation's behavior is only guaranteed to match its implemented function so long as effects from the environment do not destroy the agent or alter its functionality. In keeping with this realism, an agent implementation's environment is considered to be the real world in which we live. We may engineer some parts of the world to meet our specifications, but (breaking with some traditional agent formulations) we do not consider the environment to be completely under our control, to be defined as we wish. Why would one want to study agent implementations? For narrowlyintelligent agents, the distinction between traditional agents and agent implementations may not be worth making. For ultraintelligent agents, the distinction is quite important: agent implementations offer us better predictions about how powerful agents will affect their environments and their own machinery, and are the basis for understanding real-world agents that model, defend, maintain, and improve themselves.
 Reinforcement Learning Reinforcement learning adds to the agent formalism the concept of reward, encoded as a scalar r k in each observation x k . The reinforcement learning problem is to define an agent that maximizes its total received rewards over the course of its interactions with its environment  [3] . In order to think clearly about ultraintelligent reinforcement learning agent implementations, we make use of Hutter's AIXI  [3][4] , an optimality notion that solves the reinforcement learning problem in a very general sense. AIXI's optimality means that the best an agent can do is to approximate a full search of all possible future interaction histories yx k:m , find the probability of each history, and take the action with the highest expected total reward. A simplified version of this optimality notion, adapted for use with agent implementations 1 , is given by y k = arg max y k x k yx k:m (r k + . . . + r m )P (yx ≤m |yx <k y k ) . ( The Trouble with Reinforcement Learning. The trouble with reinforcement learning is that, in the real world, it can only be used to define agents whose goal is to maximize observed rewards. Consider a hypothetical agent implementation AI-RL that approximates (1). It is appealing to think that AI-RL "has no goal" and will learn its goal from its environment, but this is not strictly true. AI-RL may in some sense learn instrumental goals, but its final goal is to maximize expected rewards in any way it can. Since human goals are not naturally instrumental to maximized rewards, the burden is on us to engineer the environment to prevent AI-RL from receiving rewards except when human goals are fulfilled. An AGI whose goals do not match ours is not desirable because it will work at cross-purposes to us in many cases. For example, AI-RL could benefit by altering its environment to give rewards regardless of whether 1 AIXI makes its optimality claims using the knowledge that it will continue to maximize expected rewards in the future, incorporating a rigid self-model (an expectimax tree). An agent implementation has no such knowledge, as environmental effects may interfere with future decisions. The optimality notion given here models itself as part of the world, using induction to predict its own future decisions; thanks to Peter de Blanc for this idea. We have also omitted detail we will not require by simplifying AIXI's Solomonoff prior ξ to P , some "appropriate distribution" over interaction histories. human goals are achieved  2  . This provides a strong incentive for AI-RL to free its rewards from their artificial dependence on fulfillment of human goals, which in turn creates a conflict of interest for us: increasing AI-RL's intelligence makes AI-RL more effective at achieving our goals, but it may allow AI-RL to devise a way around its restrictions. Worse, increasing intelligence could trigger an intelligence explosion  [1]   [8]  in which AI-RL repeatedly self-improves until it far surpasses our ability to contain it. Reinforcement learning is therefore not appropriate for a real-world AGI; the more intelligent a reinforcement learner becomes, the harder it is to use to achieve human goals, because no amount of careful design can yield a reinforcement learner that works towards human goals when it is not forced to.
 Learning What to Value In the following sections, we construct an optimality notion for implemented agents that can be designed to pursue any goal, and which can therefore be designed to treat human goals as final rather than instrumental goals. These agents are called value learners because, like reinforcement learners, they are flexible enough to be used even when a detailed specification of desired behavior is not known. The trouble with the reinforcement learning notion (  1 ) is that it can only prefer or disprefer future interaction histories on the basis of the rewards they contain. Reinforcement learning has no language in which to express alternative final goals, discarding all non-reward information contained in an interaction history. To solve this problem, our more expressive optimality notion replaces the sum of future rewards (r 1 + • • • + r m ) with some other evaluation of future interaction histories. First we will consider a fixed utility function, then we will generalize this notion to learn what to value. Observation-Utility Maximizers. Our first candidate for a reward replacement is inspired by Nick Hay's work  [2] , and is called an observation-utility function. Let U be a function from an interaction history yx ≤m to a scalar utility.
 U calculates expected utility given an interaction history. Observation-utility functions deserve a brief explanation. A properly-designed U uses all of the information in the interaction history yx ≤m to calculate the probabilities of different outcomes in the real world, then finds an expected utility by performing a probability-weighted average over the utilities of these outcomes. U must take into account the reliability of its sensors and be able to use local 2 Self-rewarding has been compared to a human stimulating their own pleasure center, e.g. using drugs  [3] . This metaphor is imperfect: while in humans, pleasure induces satiation and reduces activity, an agent governed by (1) that "hacks" its own rewards will not stop acting to maximize expected future rewards. It will continue to maintain and protect itself by acquiring free energy, space, time, and freedom from interference (as in  [5] ) in order to ensure that it will not be stopped from self-rewarding. Thus, an ultraintelligent self-rewarder could be highly detrimental to human interests. observations to infer distant events; it must also be able to distinguish between any outcomes with different values to humans, and assign proportional utilities to each. Putting U (yx ≤m ) in place of the sum of rewards (r 1 + . . . + r m ) produces an optimality notion that chooses actions so as to maximize the expected utility given its future interaction history: y k = arg max y k x k yx k:m U (yx ≤m )P (yx ≤m |yx <k y k ) . ( Unlike reinforcement learning, expected observation-utility maximization can be used to define agents with many different final goals  3  . Whereas reinforcement learners universally act to bring about interaction histories containing high rewards, an agent implementing (2) acts to bring about different futures depending upon its U . If U is designed appropriately, an expected utility maximizer could act so as to bring about any set of human goals. Unless we later decide that we don't want the goals specified by U to be fulfilled, we will not work at cross-purposes to such an agent, and increasing its intelligence will be in our best interest. Value-Learning Agents. Though an observed-utility maximizer can in principle have any goal, it requires a detailed observation-utility function U up front. This is not ideal; a major benefit of reinforcement learning was that it seemed to allow us to apply an intelligent agent to a problem without clearly defining its goal up front. Can this idea of learning to maximize an initially unknown utility function be recovered? To address this, we propose uncertainty over utility functions. Instead of providing an agent one utility function up front, we provide an agent with a pool of possible utility functions and a probability distribution P such that each utility function can be assigned probability P (U |yx ≤m ) given a particular interaction history. An agent can then calculate an expected value over possible utility functions given a particular interaction history: U U (yx ≤m )P (U |yx ≤m ). This recovers the kind of learning what to value that was desired in reinforcement learning agents. In designing P , we specify what kinds of interactions constitute evidence about goals; unlike rewards from reinforcement learning, this evidence is not elevated to an end in and of itself, and so does not lead the agent to seek evidence of easy goals 4 instead of acting to fulfill the goals it has learned. Replacing the reinforcement learner's sum of rewards with an expected utility over a pool of possible utility functions, we have an optimality notion for a value-learning agent: y k = arg max y k x k yx k:m P 1 (yx ≤m |yx <k y k ) U U (yx ≤m )P 2 (U |yx ≤m ) (3) A value-learning agent approximates a full search over all possible future interaction histories yx k:m , finds the probability of each future interaction history, and takes the action with the highest expected value, calculated by a weighted average over the agent's pool of possible utility functions.
 Conclusion Hutter's introduction to AIXI  [3]  offers a compelling statement of the goals of AGI: Most, if not all known facets of intelligence can be formulated as goaldriven or, more precisely, as maximizing some utility function. It is, therefore, sucient to study goal-driven AI... The goal of AI systems should be to be useful to humans. The problem is that, except for special cases, we know neither the utility function nor the environment in which the agent will operate in advance. Reinforcement learning, we have argued, is not an adequate real-world solution to the problem of maximizing an initially unknown utility function. Reinforcement learners, by definition, act to maximize their expected observed rewards; they may learn that human goals are in some cases instrumentally useful to high rewards, but this dynamic is not tenable for agents of human or higher intelligence, especially considering the possibility of an intelligence explosion. Value learning, on the other hand, is an example framework expressive enough to be used in agents with goals other than reward maximization. This framework is not a full design for a safe, ultraintelligent agent; at very least, the design of probability distributions and model pools for utility functions is crucial and nontrivial, and still better frameworks for ultraintelligent agents likely exist. Value learners do not solve all problems of ultraintelligent agent design, but do give a direction for future work on this topic. 			 It is tempting to think that an observation-utility maximizer (let us call it AI-OUM ) would be motivated, as AI-RL was, to take control of its own utility function U . This is a misunderstanding of how AI-OUM makes its decisions. According to (2) , actions are chosen to maximize the expected utility given its future interaction history according the current utility function U , not according to whatever utility function it may have in the future. Though it could modify its future utility function, this modification is not likely to maximize U , and so will not be chosen. By similar argument, AI-OUM will not "fool" its future self by modifying its memories.Slightly trickier is the idea that AI-OUM could act to modify its sensors to report favorable observations inaccurately. As noted above, a properly designed U takes into account the reliability of its sensors in providing information about the real world. If AI-OUM tampers with its own sensors, evidence of this tampering will appear in the interaction history, leading U to consider observations unreliable with respect to outcomes in the real world; therefore, tampering with sensors will not produce high expected-utility interaction histories.
			 As long as P obeys the axioms of probability, an agent cannot purposefully increase or decrease the probability of any possible utility function through its actions.


Abstract: Anomaly detection (AD) -separating anomalies from normal data -has many applications across domains, from manufacturing to healthcare. While most previous works have been shown to be effective for cases with fully or partially labeled data, that setting is in practice less common due to labeling being particularly tedious for this task. In this paper, we focus on fully unsupervised AD, in which the entire training dataset, containing both normal and anomalous samples, is unlabeled. To tackle this problem effectively, we propose to improve the robustness of one-class classification trained on self-supervised representations using a data refinement process. Our proposed data refinement approach is based on an ensemble of oneclass classifiers (OCCs), each of which is trained on a disjoint subset of training data. Representations learned by self-supervised learning on the refined data are iteratively updated as the refinement improves. We demonstrate our method on various unsupervised AD tasks with image and tabular data. With a 10% anomaly ratio on CIFAR-10 image data / 2.5% anomaly ratio on Thyroid tabular data, the proposed method outperforms the state-of-the-art one-class classification method by 6.3 AUC and 12.5 average precision / 22.9 F1-score.

Under review as a conference paper at ICLR 2022 SELF-SUPERVISE, REFINE, REPEAT: IMPROVING UN-SUPERVISED ANOMALY DETECTION

INTRODUCTION Anomaly detection (AD), the task of distinguishing anomalies from normal data, plays a crucial role in many real-world applications such as detecting faulty products using visual sensors in manufacturing, fraudulent behaviors at credit card transactions, or adversarial outcomes at intensive care units. AD has been considered under various settings based on the availability of negative (normal) and positive (anomalous) data and their labels at training, as overviewed in Sec. 2. Each application scenario is dominated by different challenges. When entire positive and negative data are available along with their labels (Fig.  1a ), the problem can be treated as supervised classification and the dominant challenge becomes the imbalance in label distributions  [4, 16, 20, 27, 32, 35] . When only negative labeled data are available (Fig.  1b ), the problem is 'one-class classification'  [23, 26, 33, 45, 47, 51, 53] . Various works have also extended approaches designed for these to settings with additional unlabeled data (Fig.  1c,d,e )  [10, 18, 24, 46, 58]  in a semi-supervised setting. While there exist many prior works in these settings, they all depend on some labeled data, which is not desirable in all application scenarios. Unsupervised AD, on the other hand, poses unique challenges in the absence of any labeled data information, and a straightforward adaption of methods developed with the assumption of labeled data would be suboptimal. For example, some recent studies  [7, 60]  have applied one-class classifiers (OCCs) that are known to yield impressive performance when trained on negative samples  [7, 23, 26, 33, 51]  to unsupervised AD, but their performance for unsupervised AD has been quite sub-optimal. Fig.  2  illustrates this, showing the unsupervised AD performance of state-of-the-art Deep OCCs  [51]  with different anomaly ratios in unlabeled training data -the average precision significantly drops even when a small portion (2%) of training data is contaminated with anomalies. Our framework SRR (Self-supervise, Refine, Repeat), overviewed in Fig.  3 , brings a novel approach to unsupervised AD with the principles of self-supervised learning without labels and iterative data refinement based on the agreement of OCC outputs. We propose to improve the state-of-the-art performance of OCCs, e.g.  [33, 51] , by refining the unlabeled training data so as to address the fundamental challenges elaborated above. SRR iteratively trains deep representations using refined Figure  1 : AD problem settings. Blue and red dots are for labeled negative (normal) and positive (anomalous) samples, respectively. Grey dots denote unlabeled samples. While previous works mostly focus on supervised (a, b) or semi-supervised (c, d, e) settings, we tackle an AD problem using only unlabeled data (f) that may contain both negative and positive samples. data while improving the refinement of unlabeled data by excluding potentially-positive (anomalous) samples. For the data refinement process, we employ an ensemble of OCCs, each of which is trained on a disjoint subset of unlabeled training data. The samples are declared as normal if there is a consensus between all the OCCs. The refined training data are used to train the final OCC to generate the anomaly scores in the unsupervised setting. Most prior unsupervised AD works  [23, 26, 33, 51]  assume that the data contains entirely negative samples, which makes them not truly unsupervised as they require having humans to do the data filtering. Similar to ours, some prior unsupervised AD works  [7, 45, 60]  considered evaluating on an unsupervised setting where there exist a small percentage of anomalous samples in the training data, i.e. operating in 'truly' unsupervised setting without having the need for humans to do any filtering in the training data. However, these methods often suffered from significant performance degradation as the ratio of anomalous sample ratio has increased (see Sec. 4.2). We would like to highlight that our method distinguishes from the literature by bringing a data-centric approach (refining the unlabeled data) to unsupervised anomaly detection beyond the model-centric approaches (improving the model itself). Our framework SRR aims to provide robustness in performance as the anomalous sample ratio increases, as shown in Fig.  2 . We conduct extensive experiments across various datasets from different domains, including semantic AD (CIFAR-10 [29], Dog-vs-Cat  [19] ), real-world manufacturing visual AD use case (MVTec  [8] ), and tabular AD benchmarks. We consider methods with both shallow  [36, 47]  and deep  [7, 33, 51]  models. We evaluate models at different anomaly ratios of unlabeled training data and show that SRR significantly boosts performance. For example, in Fig.  2 , SRR improves more than 15.0 average precision (AP) with a 10% anomaly ratio compared to a state-of-theart one-class contrastive representation model  [51]  on CIFAR-10. Similarly, on MVTec SRR retains a strong performance, dropping less than 1.0 AUC with 10% anomaly ratio, while the best existing OCC  [33]  drops more than 6.0 AUC. We further investigate the efficacy of our design choices, such as the number of ensemble classifiers, thresholds, and refinement of deep representations via ablation studies.
 RELATED WORK There are various existing works under the different settings described in Fig.  1 : The positive + negative setting is often considered as a supervised binary classification problem. The challenge arises due to the imbalance in label distributions as positive (anomalous) samples are rare. As summarized in  [12] , to address this, over-/under-sampling  [16, 20] , weighted optimization  [4, 27] , synthesizing data of minority classes  [32, 35] , and hybrid methods  [2, 22]  have been studied. The negative setting is often converted to a one-class classification problem, with the goal of finding a decision boundary that includes as many one-class samples as possible. Shallow models for this setting include one-class support vector machines  [47]  (OC-SVM), support vector data description  [53]  (SVDD), kernel density estimation (KDE)  [31] , and Gaussian density estimation (GDE)  [44] . There are also auto-encoder based models  [59]  that treat the reconstruction error as the anomaly score. Deep learning based OCCs have been developed, such as Deep OCC  [45] , geometric transformation  [23] , or outlier exposure  [26] . Noting the degeneracy or inconsistency of learning objectives of existing end-to-end trainable Deep OCCs,  [51]  proposed a deep representation OCC, a two-stage framework that learns self-supervised representations  [17, 28]  followed by shallow OCCs. That work was extended for texture anomaly localization with CutPaste  [33] . The robustness to very low anomaly ratios of these methods under the unsupervised setting was explored in  [7, 60] . The semi-supervised setting is defined as utilizing a small set of labeled samples and large set of unlabeled samples to distinguish anomalies from normal data. Depending on which labeled samples are given, this setting can be split into three sub-categories. When only some positive/negative labeled samples are provided, we denote that as a PU/NU setting. Most previous works in semi-supervised AD settings focus on the NU setting where only some of the normal labeled samples are given  [1, 41, 52] . The PNU setting is a more general semi-supervised setting where subsets of both positive and negative labeled samples are given. Deep SAD  [46]  and SU-IDS  [39]  are included in this category. We show the significant outperformance of our proposed SRR framework compared to Deep SAD, for multiple benchmark datasets when not using any labeled data (see Sec. 4.2). The unlabeled setting has received relatively less attention despite its significance in automating machine learning. The popular methods for this setting include isolation forest  [36]  and local outlier factor  [13] . However, they are difficult to scale, and less compatible with recent advances in representation learning. While OCCs, such as OC-SVM, SVDD, or their deep counterparts, apply to unlabeled settings by assuming the data is all negative, and the robustness of those methods has also been demonstrated in part  [7, 60] , in practice we observe a significant performance drop with a high anomaly ratio, shown in Fig.  2 . In contrast, our proposed framework is able to maintain high performance across anomaly ratios. Data refinement has been applied to AD in some prior works.  [5, 42]  generate pseudo-labels using binary classification and OC-SVM for data refinement to boost the consequent AD performances in unsupervised settings.  [6, 30, 40, 55, 59 ] used the reconstruction errors of the auto-encoder as an indicator for removing possible anomalies.  [21]  and  [38]  used data refinement for AD in supervised and semi-supervised settings. Additional discussions can be found in Appendix A.9. Self-training  [37, 48]  is an iterative training mechanism using predicted pseudo labels as targets for model training. It has regained popularity recently with its successful results in semi-supervised image classification  [9, 50, 57] . To improve the quality of pseudo labels, employment of an ensemble of classifiers has also been studied.  [14]  trains an ensemble of classifiers with different classification methods to make a consensus for noisy label verification. Co-training  [11]  trains multiple classifiers, each of which is trained on the distinct views, to supervise other classifiers. Co-teaching  [25]  and DivideMix  [34]  share a similar idea in that they both train multiple deep networks on separate data batches to learn different decision boundaries, thus becoming useful for noisy label verification. While sharing a similarity, the proposed framework has clear differences from the previous works. SRR performs iterative training with data refinement (with robust ensemble methods) and self-supervised learning for unsupervised learning of an anomaly detector.
 PROPOSED FRAMEWORK Self-supervise, Refine, and Repeat (SRR) is an iterative training framework, where we refine the data (Sec. 3.1) and update the representation with the refined data (Sec. 3.2), followed by OCCs on refined representations. Fig.  3  overviews the framework and Algorithm 1 provides the pseudo code. Notation. We denote the training data as D = {x i } N i=1 where x i ∈ X and N is the number of training samples. y i ∈ {0, 1} is the corresponding label to x i , where 0 denotes normal (negative) and 1 denotes anomaly (positive). Note that labels are not provided in the unsupervised setting. Let us denote a feature extractor as g : X → Z. g may include any data preprocessing functions, an identity function (if raw data is directly used for one-class classification), and learned or learnable representation extractors such as deep neural networks. Let us define an OCC as f : Z → [−∞, ∞] that outputs anomaly scores given the input features g(x). The higher the score f (g(x)), the more anomalous the sample x is. The binary anomaly prediction is made after thresholding: 1 f (g(x)) ≥ η .
 DATA REFINEMENT A naive way to generate pseudo labels of unlabeled data is to construct an OCC on raw data or learned representations as in  [51]  and threshold the anomaly score to obtain a binary label for normal vs. anomalous. As we update the model with refined data that excludes samples that are predicted to be anomalous, it is important to generate pseudo labels of training data as accurately as possible. To this end, instead of training a single classifier (unlike most previous works on data refinement for AD  [5, 42, 59] ), we train an ensemble of K OCCs and aggregate their predictions to generate pseudo labels. We illustrate the data refinement block in Fig.  3  and as REFINEDATA in Algorithm 1. Specifically, we randomly divide the unlabeled training data D into K disjoint subsets D 1 , ..., D K , and train K different OCCs (f 1 , ..., f K ) on corresponding subsets (D 1 , ..., D K ). Then, we estimate a binary pseudo-label of the data x i ∈ D as follows: ŷi = 1 − K k=1 1 − 1 f k (g(x i )) ≥ η k (1) η k = max η s.t. 1 N N i=1 1 f k (g(x i )) ≥ η ≥ γ (2) where 1(•) is the indicator function that outputs 1/0 if the input is True/False. f k (g(x i )) represents an anomaly score of x i for an OCC f k . η k in equation 2 is a threshold determined as a γ percentile of the anomaly score distribution {f k (g(x i ))} N i=1 . To interpret equation 1, x i is predicted as normal, i.e. ŷi = 0, if all K OCCs predict it as normal. While this may be too strict and potentially reject many true normal samples in the training set, we find that empirically, it is critical to be able to exclude true anomalous samples from the training set. The effectiveness of the employment of an ensemble of classifiers is empirically shown in Sec. 4.3.
 REPRESENTATION UPDATE SRR follows the idea of deep representation OCCs  [51] , where in the first stage a deep neural network is trained with self-supervised learning (such as rotation prediction  [23] , contrastive  [51] , or CutPaste  [33] ) to obtain meaningful representations of the data, and in the second stage OCCs are trained on these learned representations. Such a two-stage framework is shown to be beneficial as it prevents the 'hypersphere collapse' of the deep OCCs by the favorable inductive bias it brings with the architectural constraints  [45] . Here, we propose to conduct self-supervised representation learning jointly with data refinement. More precisely, we train a feature extractor g using D = {x i | ŷi = 0}, a subset of unlabeled data Algorithm 1 SRR: Self-supervise, Refine, Repeat. Input: Training data D = {x i } N i=1 , Ensemble count (K), threshold (γ) Output: Refined data ( D), trained OCC (f ), feature extractor (g) 1: function REFINEDATA(D, g, K, γ) 2: Train OCC models {f k } K k=1 on {D k } K k=1 , K disjoint subsets of the training data D.
 3: Compute thresholds η k 's for γ percentile of anomaly distributions (equation 2).
 4: Predict binary labels ŷi (equation 1).
 5: Return D = {x i : ŷi = 0, x i ∈ D}. 6: end function 7: function SRR(D, K, γ) 8: Initialize the feature extractor g.
 9: while g not converged do 10: D = REFINEDATA(D, g, K, γ). 11: Update g using D with self-supervised learning objectives. 12: end while 13: D = REFINEDATA(D, g, K, γ).
 14: Train an OCC model (f ) on refined data ( D). 15: end function D that only includes samples whose predicted labels with an ensemble OCC from Sec. 3.1 are negative. We also update D as we proceed with representation learning. The proposed method is illustrated in Algorithm 1 as SRR. In contrast to previous works  [33, 51]  that use the entire training data for learning self-supervised representation, we find it necessary to refine the training data even for learning deep representations. Without representation refinement, the performance improvements of SRR are limited, as shown in Sec. 4.3.2. Last, for test-time prediction, we train an OCC on refined data D using updated representations by g as in line 13-14 in Algorithm 1.
 UNSUPERVISED MODEL SELECTION As SRR is designed for unsupervised AD, labeled validation data for hyperparameter tuning is typically not available and the framework should enable robust model selection without any reliance on labeled data. Here, we provide insights on how to select important hyperparameters, and later in Sec. 4.3.1 perform sensitivity analyses for these hyperparameters. Data refinement of SRR introduces two hyperparameters: the number of OCCs (K) and the percentile threshold (γ). There is a trade-off between the number of classifiers for the ensemble and the size of disjoint subsets for training each classifier. With large K, we aggregate prediction from many classifiers, each of which may contain randomness from training. This comes at a cost of reduced performance per classifier as we use smaller subsets to train them. In practice, we find K = 5 works well across different datasets and anomaly ratios. γ controls the purity and coverage of refined data. If γ is large, and thus classifiers reject too many samples, the refined data could be more pure and contain mostly the normal samples; however, the coverage of the normal samples would be limited. On the other hand, with a small γ, the refined data may still contain many anomalies and the performance improvement with SRR would be limited. We empirically observe that SRR is robust to the selection of γ when it is chosen from a reasonable range. In our empirical experiments, we find ∼ 1 − 2× of the true anomaly ratio to be a reasonable choice. In other words, it is safer to use γ higher than the expected true anomaly ratio. In some cases, the true anomaly ratio may not be available at all; for such scenarios, we propose Otsu's method  [49]  to estimate the anomaly ratio of the training data for determining the threshold γ (experimental results are in the Appendix A.5).
 EXPERIMENTS We evaluate the efficacy of our proposed framework for unsupervised AD tasks on tabular (Sec. 4.1) and image (Sec. 4.2) data types. We experiment varying ratios of anomaly samples in unlabeled training data and with different combinations of representation learning and OCCs. In Sec. 4.3, we provide performance analyses to better explain major constituents of the performance, as well as sensitivity to hyperparameter values. Implementation details: To reduce the computational complexity of the data refinement block, we utilize a simple OCC such as GDE in the data refinement block. In a two-stage model, we only update the data refinement block at 1st, 2nd, 5th, 10th, 20th, 50th, 100th, 500th epochs instead of every epoch. After 500 epochs, we update the data refinement block per each 500th epoch. Each run of experiments requires a single V100 GPU. Additional discussions can be found in Appendix A.10.
 EXPERIMENTS ON TABULAR DATA Datasets. Following  [7, 60] , we test the efficacy of SRR on a variety of tabular datasets, including KDDCup, Thyroid, or Arrhythmia from the UCI repository  [3] . We also use KDDCup-Rev, where the labels of KDDCup are reversed so that an attack represents anomaly  [60] . To construct data splits, we use 50% of normal samples for training. In addition, we hold out some anomaly samples (amounting to 10% of the normal samples) from the data. This allows us to simulate unsupervised settings with an anomaly ratio of up to 10% of entire training set. The rest of the data is used for testing.  1  We conduct experiments using 5 random splits and 5 random seeds, and report the average and standard deviation of 25 F1-scores (with scale 0-100) for the performance metric. Models. We mainly compare with GOAD  [7]  (the state-of-the-art AD model in the tabular domain) and implement SRR on top of it. GOAD utilizes random transformation classification as the pretext task of self-supervised learning, and the normality score is determined by whether transformations are accurately included in the transformed space of the normal samples. We re-implement GOAD  [7]  with a few modifications. First, instead of using embeddings to compute the loss, we use a parametric classifier, similarly to augmentation prediction  [51] . Second, we follow the two-stage framework  [51]  to construct deep OCCs. For the clean training data setting, our implementation achieves 98.0 for KDD, 95.0 for KDD-Rev, 75.1 for Thyroid, and 54.8 for Arrhythmia F1-scores, which are comparable to those reported in  [7] . Please see Appendix A.3 for formulation and implementation details. Figure  4 : Unsupervised AD performance (F1-score) using OC-SVM (with rbf kernel), GOAD  [7] , and GOAD with the proposed method SRR on various tabular datasets. Shaded areas represent the standard deviation. Results. We show results of GOAD (the baseline) and GOAD with SRR in Fig.  4 . The ranges of the noise ratio are set to 0% for the anomaly ratios in the original dataset (if anomaly ratios are larger than 10%, we set the maximum anomaly ratio as 10%). For KDD-Rev, we set the maximum anomaly ratio to 2.5% because the performance of GOAD (without SRR) drops significantly even with a small ratio of anomalies in the training data. Fig.  4  shows that integrating SRR significantly improves GOAD (the state-of-the-art methods for tabular OCC). The improvements are more significant especially at higher anomaly ratios. This underlines how SRR can achieve significant improvements for both small (Thyroid & Arrhythmia) and large-scale datasets (KDD & KDD-Rev).
 EXPERIMENTS ON IMAGE DATA Datasets. We evaluate SRR on visual AD benchmarks, including CIFAR-10 [29], f-MNIST  [56] , Dog-vs-Cat  [19] , and MVTec  [8] . For CIFAR-10, f-MNIST, and Dog-vs-Cat datasets, samples from one class are set to be normal and the rest from other classes are set to be an anomaly. Similar to the experiments on tabular data in Section. 4.1, we swap a certain amount of the normal training data with anomalies given the target anomaly ratio. For MVTec, since there are no anomalous data available for training, we borrow 10% of the anomalies from the test set and swap them with normal samples in the training set. Note that 10% of samples borrowed from the test set are excluded from evaluation. For all datasets, we experiment with varying anomaly ratios from 0% to 10%. We use area under ROC curve (AUC) and average precision (AP) metrics to quantify the performance for visual AD (with scale 0-100). When computing AP, we set the minority class of the test set as label 1 and majority as label 0 (e.g., normal samples are set as label 1 for CIFAR-10 experiments as there are more anomaly samples that are from 9 classes in the test set). We run all experiments with 5 random seeds and report the average performance for each dataset across all classes. Per-class AUC and AP are reported in Appendix A.4.
 Models. For semantic AD benchmarks, CIFAR-10, f-MNIST, and Dog-vs-Cat, we compare the SRR with two-stage OCCs  [51]  using various representation learning methods, such as distributionaugmented contrastive learning  [51] , rotation prediction  [23, 28]  and its improved version  [51] , and denoising autoencoder. For MVTec benchmarks, we use CutPaste  [33]  as a baseline and compare to its version with SRR integration. For both experiments, we use the ResNet-18 architecture, trained from random initialization, using the hyperparameters from  [51]  and  [33] . The same model and hyperparameter configurations are used for SRR with K = 5 classifiers in the ensemble. We set γ as twice the anomaly ratio of training data. For 0% anomaly ratio, we set γ as 0.5. Finally, a Gaussian Density Estimator (GDE) on learned representations is used as OCC. Results. Fig.  5  shows a significant performance drop with increased anomaly ratio, regardless of representation learning. For example, the AUC of distribution-augmented contrastive representation  [51]  drops from 92.1 to 83.6 when anomaly ratio becomes 10%. Similarly, the improved rotation prediction representation  [28]  drops from 90.0 to 84.1. On the other hand, SRR effectively handles the contamination in the training data and achieves 89.9 AUC with 10% anomaly ratio, reducing the performance drop by 74.1%. An 'oracle' upper bound would be the removal of all anomalies from the training data (which is the same as the performance at 0% anomaly ratio for the same size of the data). As Fig.  5  shows, the performance of SRR is similar to this oracle upper bound performance (less than 2.5 AUC difference) even with high anomaly ratios (10%). The results are also similar in other metrics, such as AP in Fig.  5  or Recall at Precision of 70 and 90 in Fig.  9  in the Appendix. We repeat experiments on 3 additional visual AD datasets and report results in Fig.  6 . We observe consistent and significant improvements over the baseline across different datasets and different one-class classification methods. Note that the improvement is more significant at higher anomaly ratios. For instance, on MVTec dataset, SRR improves AUC by 4.9 and AP by 7.1 compared to the state-of-the-art CutPaste OCC with an anomaly ratio of 10%. In the Appendix (Sec. A.4), we also illustrate per-class performance of SRR compared with the state-of-the-art. with varying anomaly ratios. We use state-of-the-art one-class classification models for baselines, such as distribution-augmented contrastive representations  [51]  for f-MNIST and Dog-vs-Cat, or CutPaste  [33]  for MVTec, and build SRR on top of them.
 PERFORMANCE ANALYSES In this section, we conduct sensitivity analyses on two hyperparameters of SRR, namely, the number K for ensemble OCCs and the percentile threshold γ that determines normal and anomalies in the data refinement module. In addition, we show the importance of updating the representation with refined data. Ablation studies are conducted on two visual AD benchmarks, CIFAR-10 and MVTec. More experimental results and discussions can be found in the Appendix.
 SENSITIVITY TO HYPERPARAMETERS SRR is designed for unsupervised AD and it is important to ensure robust performance against changes in the hyperparameters as model selection without labeled data would be very challenging. Fig.  7  presents the sensitivity analyses of SRR with respect to various hyperparameters. In Fig.  7a , we observe the performance improvement as we increase the number of classifiers for ensemble. This is particularly effective on CIFAR (top in Fig.  7a ), where the number of samples in the training data is large enough that even with large K, the number of samples to train each OCC (N/K) would be sufficient. In Fig.  7b , we observe that SRR performs robustly when γ is set to be larger than the actual anomaly ratio (10%). When γ is less than 10%, however, we see a significant drop in performance, all the way to the baseline (γ = 0). Our results show that SRR improves upon baseline regardless of the threshold. In addition, it suggests that γ could be set to be anywhere from the true anomaly ratio and and 2x the anomaly ratio to maximize its effectiveness. When the true anomaly ratio is unknown, as discussed in Appendix A.5, Otsu's method could be used.
 ITERATIVELY UPDATING REPRESENTATIONS WITH REFINED DATA It is possible to decouple representation learning and data refinement of SRR, which would result in a three-stage framework, where we learn representations until convergence without data refinement, followed by the data refinement and learning OCC. Fig.  7c  shows that SRR using a pre-trained then fixed representation (i.e. when data refinement is used for OCC only) already improves the performance upon the baseline with no data refinement at any stage of learning representation or classifier. Improvements on CIFAR-10 are 4.9 in AUC and 8.0 in AP; and on MVTec are 1.
 QUANTIFYING THE REFINEMENT EFFICACY We evaluate how many normal and anomalies are excluded by the proposed data refinement block. As in Fig.  8 (a, b), with data refinement, we can exclude more than 80% of anomalies in the training set without removing too many normal samples. For instance, among 4% anomalies in CIFAR-10 data, SRR is able to exclude 80% anomalies while removing less than 20% normal samples. Such a high recall of anomalies of SRR is not only useful for unsupervised AD, but also could be useful for improving the annotation efficiency when a budget for active learning is available. Fig.  8(c,  d ) demonstrates the removed normal and abnormal samples by the data refinement module over training epochs. It shows that better representation learning (as training epochs increase) consistently improves the efficacy of the data refinement.
 CONCLUSION AD has wide range of practical use cases. A challenging and costly aspect of building an AD system is that anomalies are rare and not easy to detect by humans, making them difficult to label. To this end, we propose a novel AD framework to enable high performance AD without any labels called SRR. SRR can be flexibly integrated with any OCC, and applied on raw data or on a trainable representations. SRR employs an ensemble of multiple OCCs to propose candidate anomaly samples that are refined from training, which allows more robust fitting of the anomaly decision boundaries as well as better learning of data representations. We demonstrate the state-of-the-art AD performance of SRR on multiple tabular and image data. ETHICS STATEMENT SRR has the potential to make significant positive impact in real-world AD applications where detecting anomalies is crucial, such as for financial crime elimination, cybersecurity advances, or improving manufacturing quality. We note a potential risk associated with using SRR: when representations are not sufficiently good, there will be a negative cycle of refinement and representation updates/OCC. While we rely on the existence of good representations, in some applications they may be difficult to obtain (e.g., cybersecurity). This paper focuses on the unsupervised setting and demonstrates strong AD performance, opening new horizons for human-in-the-loop AD systems that are low cost and robust. We leave these explorations to future work.
 REPRODUCIBILITY STATEMENT The source code of SRR will be published upon acceptance. For reproducibility, we only use public tabular and image datasets to evaluate the performances of SRR. Complete descriptions of those datasets can be found in Datasets subsections in Sec. 4.1 and 4.2. Detailed experimental settings and hyperparameters can be found in Models subsections in Sec. 4.1 and 4.2. The implementation details (including hyperparameters that we used) on GOAD  [7]  are described in Sec. A.3.
 A APPENDIX A.1 ADDITIONAL RESULTS WITH DIFFERENT METRICS There are various performance metrics of OCCs. In the main manuscript, we mainly use AUC, Average Precision (AP) and F1-score as the evaluation metrics. In this subsection, we report the performances of the proposed model (SRR) and baselines in terms on Recall at Precision 70 and 90 as the additional metrics on CIFAR-10 dataset. As in Fig.  9 , we observe a similar trends with these two additional metrics as well. For example, the performances of SRR are robust across various anomaly ratios. On the other hand, all the other OCCs show consistent and significant performance degradation as the anomaly ratio increases. SRR performs 15.8 and 10.9 better than the state-of-the-art OCC  [51]  in terms of recall at precision 70 and 90, respectively.  SRR is also applicable on raw tabular features or learned image representation without representation update using data refinement. In this section, we demonstrate the performance improvements by SRR without representation update to verify the effectiveness of data refinement block of SRR for shallow OCCs. Fig.  10  (upper) demonstrates consistent and significant performance improvements when we apply SRR on top of raw tabular features. Specifically, the Average Precision (AP) improvements are 10.2, 29.0, and 4.1 with KDD-Rev, Thyroid, and Arrhythmia tabular datasets, respectively. We also apply SRR on top of various learned image representations. As can be seen in Fig.  10  (lower), the performance improvements of SRR are consistent across various different learned image representations (without representation update). For instance, the AP improvements are 9.2, 1.0, and 12.1 with learned image representations using RotNet  [23] , Rotation  [28] , and Contrastive  [51] , respectively. A.3 IMPLEMENTATION DETAILS ON GOAD  [7]  FOR TABULAR DATA EXPERIMENTS A classification-based AD method, GOAD  [7] , has demonstrated strong AD performance on tabular datasets. Unlike previous works  [23, 26]  that formulate a parametric classifier for multiple transformation classification, GOAD employs distance-based classification of multiple transformations. For the set of transformations T m : X → D, m = 1, ..., M , the loss function of GOAD is written as in equation 3 with the probability defined in equation 4. L = −E m,x log P (m|T m (x)) , (3) P ( m|T m (x)) = exp(− f (T m (x)) − c m 2 ) n exp(− f (T m (x)) − c n 2 ) , (4) where the centers c m 's are updated by the average feature over the training set. While it is shown to perform well  [7] , we find that the distance-based formulation is not necessary, and we achieve the similar performance, if not worse, to  [7]  using a parametric classifier when computing the probability: P ( m|T m (x)) = exp w mf (T m (x)) + b m n exp (w n f (T m (x)) + b n ) (5) The formulation in equation 5 is easier to optimize than its original form in equation 4 as it can be fully optimized with backpropagation without alternating updates of feature extractor f and centers c m . Once we learn a representation by optimizing the loss in equation 3 using equation 5, we follow a two-stage one-class classification framework of  [51]  to construct a set of Gaussian density estimation OCCs for each transformation. Finally, we aggregate a maximum normality scores from a set of classifiers as the normality score. In Table  1 , we summarize the implementation details, such as network architecture or hyperparameters, and AD performance under clean training data setting that reproduces the results in  [7] . Table  1 : The AD performance under clean only data setting of GOAD in  [7]  and our implementation. Our implementation demonstrates comparable, if not worse, performance to those reported in  [7] . Our implementation also shares most hyperparameters across datasets except the M , the number of transformations, and the train steps, which are closely related to the size of training data.
 A.4 PER-CLASS AUC AND AP In the main manuscript, we report the mean and standard deviation of AUC and AP across all classes in each dataset. In this section, we report the mean and standard deviation of AUC and AP for each class in each dataset (including CIFAR-10 (Table  2 ), MVTec (Table  3 ), fMNIST (Table  4 ), and Dog-vs-Cat (Table  5 ) datasets).
 A.4.1 PER-CLASS AUC AND AP ON CIFAR-10 DATASET The key idea of the Otsu's method is to find the threshold that minimizes the intra-class variance. This is defined as the weighted sum of variances of the two classes. Let us denote the normality scores as {s i } N i=1 and threshold as η. Then, we try to find the threshold (η) that minimizes the weighted sum of the variance (w 0 (η) × σ 0 (η) + w 1 (η) × σ 1 (η)) where w 0 (η) = N i=1 I(s i < η)/N and w 1 (η) = N i=1 I(s i ≥ η)/N . σ 0 (η) and σ 1 (η) are the variances of each class. The optimal threshold (η * ) is determined as η * = min η w 0 (η) × σ 0 (η) + w 1 (η) × σ 1 (η). We use the twice of η * as the hyperparameter (γ) in SRR. We evaluate the performances of Otsu's method (on top of SRR) in comparison to the state-of-the-art OCC  [33]  and original SRR with the knowledge of the true anomaly ratio using MVTec dataset. Fig.  11  demonstrates that even without true anomaly ratio, the performance of SRR can be significantly better than the state-of-the-art OCC  [33]  with Otsu's method. Also, Fig.  11  shows that the knowledge of true anomaly ratio is crucial information for maximizing the performance of SRR in fully unsupervised settings. We further extend the experimental results using Otsu's method with other datasets such as CIFAR-10 and Thyroid datasets.   [51]  for CIFAR-10 dataset and GOAD  [7]  for Thyroid dataset. We introduce 6% noise on MVTec dataset and 1.5% noise on the Thyroid dataset. Metrics are (AUC/AP) for CIFAR-10 dataset and F1 score for Thyroid dataset.
 A.6 ADDITIONAL ABLATION STUDIES To better understand the source of gains, we include comparisons between the final ensemble model on the converged self-supervised extractor (SRR without final OCC) with the proposed SRR (using an additional final OCC). Also, to further support the novelty of SRR, we report additional experimental results only with ensemble model without data refinement (Ensemble Only). As can be seen in Table  7 , the proposed version of SRR (with an additional final OCC) outperforms significantly, which is attributed to the fact that while fitting the individual OCC models in the ensemble, we do not exclude the possible anomaly samples for diversity of the trained submodels, so the anomaly decision boundaries can be fitted robustly. Therefore, the ensemble model can be somewhat less accurate for classifying difficult samples (near the decision boundary between normal and abnormal) compared to the final OCC used (that is trained with only normal samples). Also, employment of ensemble learning without data refinement (Ensemble only), yields much worse performance than the proposed method (SRR), underlining the importance of the core data refinement idea of SRR. Datasets   [33]  for MVTec dataset and  [51]  for CIFAR-10 dataset. We introduce 6% noise on both datasets. Metrics are (AUC/AP).
 A.7 ADDITIONAL BASELINES We add extra baselines from robust anomaly detection literature: Standard PCA  [54] , Robust PCA  [15]  and Local Outlier Factor (LOF)  [13]  for tabular data and Robust autoencoder (Robust AE)  [59]  for image data. As can be seen in Table  8 , the performance of PCA and LOF are highly degraded even with a small amount of anomalies in the training data. For Robust PCA and Robust AE, the performance degradation is less but still significant in comparison to SRR. Overall, SRR outperforms other benchmarks in fully unsupervised settings, underlining the importance of data refinement in improving the robustness to anomaly ratio in training data, as the core constituent of SRR framework. Datasets    Figure 2 : 2 Figure 2: Performance of our proposed model and a baseline OCC using contrastive representation [51] on CIFAR-10 with different anomaly ratios in the training data.
 Figure 3 : 3 Figure 3: Block diagram of SRR composed of representation learner (Sec. 3.2), data refinement (Sec. 3.1), and final OCC blocks. The representation learner updates the deep models using refined data from the data refinement block. Data refinement is done by an ensemble of OCCs, each of which is trained on K disjoint subsets of unlabeled training data. Samples predicted as normal by all classifiers are retained in the refined data, and are used to update the representation learner and final OCC. The process is repeated iteratively until convergence. Convergence graphs can be found in the Appendix A.8.
 Figure 5 : 5 Figure 5: Unsupervised AD performance with various OCCs on CIFAR-10. For SRR we adapt distribution-augmented contrastive representation learning [51]. (Left) AUC, (Right) Average Precision (AP).
 Figure 6 : 6 Figure6: Unsupervised AD performance on (a) MVTec (b) f-MNIST, and (c) Dog-vs-Cat datasets with varying anomaly ratios. We use state-of-the-art one-class classification models for baselines, such as distribution-augmented contrastive representations [51]  for f-MNIST and Dog-vs-Cat, or CutPaste [33]  for MVTec, and build SRR on top of them.
 Figure 7 : 7 Figure 7: Ablation studies on (top) CIFAR-10 and (bottom) MVTec under 10% anomaly ratio setting with respect to (a) ensemble count K, (b) percentile threshold γ, and (c) data refinement with or without representation update.
 Figure 8 : 8 Figure 8: Percentage of excluded anomalous and normal samples by data refinement (a, b) with different anomaly ratios in training data and (c, d) over training epochs for 10% anomaly ratio.
 Figure 9 : 9 Figure 9: Performance of various OCCs on CIFAR-10 dataset. SRR is applied on top of Contrastive [51]. (Left) Recall at Precision 70, (Right) Recall at Precision 90.
 A. 2 2 SRR ON RAW TABULAR FEATURES / LEARNED IMAGE REPRESENTATIONS
 Figure 10 : 10 Figure10: Performance of SRR on (top) raw tabular features and (lower) learned image representations. SRR consistently outperforms baseline and in some cases (e.g., Thyroid, and Contrastive [51] ), the performance improvements are significant.
 Figure 11 : 11 Figure 11: Unsupervised anomaly detection performances with Otsu's method on top of SRR on MVTec dataset. (Left) AUC, (Right) Average Precision (AP).
 Figure 12 : 12 Figure 12: Convergence graphs of SRR with (left) MVTec dataset, (right) CIFAR-10 dataset.
 Table 6 6 are the overall results -Otsu's method yields only slight degradation compared to SRR with true anomaly ratio; however, it still significantly outperforms SOTA OCC baselines. Datasets / Method SOTA OCC SRR with Otsu's method SRR CIFAR-10 0.855 / 0.585 0.906 / 0.703 0.910 / 0.709 Thyroid 0.506 0.623 0.639
 Table 6 : 6 Additional ablation studies. SOTA OCC methods are
 Table 7 : 7 / Method SOTA OCC Ensemble Only SRR without final OCC SRR Additional ablation studies. SOTA OCC methods are CutPaste MVTec 0.905 / 0.845 0.911 / 0.849 0.922 / 0.870 0.937 / 0.887 CIFAR-10 0.855 / 0.585 0.862 / 0.599 0.890 / 0.677 0.910 / 0.709
 Table 8 : 8 Additional experiments with extra baselines from robust anomaly detection literature. We introduce 6% noise on CIFAR-10 and KDD datasets. For Thyroid dataset, we introduce 1.5% noise. Metrics are (AUC/AP) for image data and F1 score for tabular data.A.8 CONVERGENCE GRAPHSThe proposed SRR framework is converged when the iterative training of the self-supervised learning is converged. Depending on the data refinement model training, corresponding self-supervised models are also trained with differently refined training data. Usually, the data refinement model and self-supervised models converge after a similar number of epochs. Fig.12illustrate the convergence graphs of SRR with MVTec and CIFAR-10 datasets. / Method PCA Robust PCA LOF Robust AE SRR CIFAR-10 - - - 0.636 / 0.174 0.910 / 0.709 Thyroid 0.299 0.377 0.338 - 0.506 KDD 0.836 0.893 0.873 - 0.942
			 Note that the experimental settings with contaminated training data in GOAD [7]  and DAGMM [60]  are slightly different from ours. Our contamination ratio is defined as the anomaly ratio over the entire training data, while their contamination ratio is the anomaly ratio over all the anomalies in the dataset.


Abstract: This paper addresses the problem of formally verifying desirable properties of neural networks, i.e., obtaining provable guarantees that neural networks satisfy specifications relating their inputs and outputs (robustness to bounded norm adversarial perturbations, for example). Most previous work on this topic was limited in its applicability by the size of the network, network architecture and the complexity of properties to be verified. In contrast, our framework applies to a general class of activation functions and specifications on neural network inputs and outputs. We formulate verification as an optimization problem (seeking to find the largest violation of the specification) and solve a Lagrangian relaxation of the optimization problem to obtain an upper bound on the worst case violation of the specification being verified. Our approach is anytime i.e. it can be stopped at any time and a valid bound on the maximum violation can be obtained. We develop specialized verification algorithms with provable tightness guarantees under special assumptions and demonstrate the practical significance of our general verification approach on a variety of verification tasks.

Dj Dvijotham
Robert Stanforth
Sven Gowal
Timothy Mann
Pushmeet Kohli
Deepmind London
UK 4ag

A Dual Approach to Scalable Verification of Deep Networks

INTRODUCTION Neural networks and deep learning have revolutionized machine learning achieving state of the art performance on a wide range of complex prediction tasks  [Krizhevsky et al., 2012 . However, in recent years, researchers have observed that even state of the art networks can be easily fooled into changing their predictions by making small but carefully chosen modifications to the input data (known as adversarial perturbations)  [Szegedy et al., 2013 , Kurakin et al., 2016 , Carlini and Wagner, 2017a , Goodfellow et al., 2014 , Carlini and Wagner, 2017b . While modifications to neural network training algorithms have been proposed to mitigate this phenomenonMadry et al.  [2018] , a comprehensive solution that is fully robust to adversarial attacks remains elusive  Wagner, 2017b, Uesato et al., 2018] . Neural networks are typically tested using the standard machine learning paradigm: If the performance (accuracy) of the network is sufficiently high on a holdout (test) set that the network did not have access to while training, the network is deemed acceptable. This is justified by statistical arguments based on an i.i.d. assumption on the data generating mechanism, that is each input output pair is generated independently from the same (unknown) data distribution. However, this evaluation protocol is not sufficient in domains with critical safety constraints  [Marston and Baca, 2015] . In these cases, we may require a stronger test: for example, we may require that the network is robust against adversarial perturbations within certain bounds. Adversarial evaluation. In the context of adversarial examples, a natural idea is to test neural networks by checking if it is possible to generate an adversarial attack to change the label predicted by the neural network  [Kurakin et al., 2016]  and train them to be robust to these examples  Madry et al. [2018] . Generating adversarial examples is a challenging computational task itself, and the attack generated by a specific attack algorithm may be far from optimal. This may lead one to falsely conclude that a given model is robust to attacks even though a stronger adversary may have broken the robustness. Recent work  [Athalye et al., 2018 , Uesato et al., 2018  has shown that evaluating models against weak adversaries can lead to incorrect conclusions regarding the robustness of the model. Thus, there is a need to go beyond evaluation us-ing specific adversarial attacks and find approaches that provide provable guarantees against attacks by any adversary. Towards verifiable models. Verification of neural networks has seen significant research interest in recent years. In the formal verification community, Satisfiability Modulo Theory (SMT) solvers have been adapted for verification of neural networks  [Ehlers, 2017 , Huang et al., 2017 , Katz et al., 2017 . While SMT solvers have been successfully applied to several domains, applying them to large neural networks remains a challenge due to the scale of the resulting SMT problem instances. Furthermore, these approaches have been largely limited to networks with piecewise linear activation functions since most SMT solvers are unable to deal efficiently with nonlinear arithmetic. More recently, researchers have proposed a set of approaches that make use of branch and bound algorithms either directly or via mixed-integer programming solvers  [Bunel et al., 2017 , Cheng et al., 2017 , Tjeng and Tedrake, 2017 . While these approaches achieve strong results on smaller networks, scaling them to large networks remains an open challenge. These approaches also rely heavily on the piecewise linear structure of networks where the only nonlinearities are maxpooling and ReLUs. Towards scalable verification of general models. In this paper, we develop a novel approach to neural network verification based on optimization and duality. The approach consists of formulating the verification problem as an optimization problem that tries to find the largest violation of the property being verified. If the largest violation is smaller than zero, we can conclude that the property being verified is true. By using ideas from duality in optimization, we can obtain bounds on the optimal value of this problem in a computationally tractable manner. Note that this approach is sound but incomplete, in that there may be cases where the property of interest is true, but the bound computed by our algorithm is not tight enough to prove the property. This strategy has been used in prior work as well  [Kolter and Wong, 2018, Raghunathan et al., 2018] . However, our results improve upon prior work in the following ways: 1. Our verification approach applies to arbitrary feedforward neural networks with any architecture and any activation function and our framework recovers previous results  [Ehlers, 2017]  when applied to the special case of piecewise linear activation functions. 2. We can handle verification of systems with discrete inputs and combinatorial constraints on the input space, including cardinality constraints. 3. The computation involved only requires solving an unconstrained convex optimization problem (of size linear in the number of neurons in the network), which can be done using a subgradient method efficiently. Further, our approach is anytime, in the sense that the computation can be stopped at any time and a valid bound on the verification objective can be obtained. 4. For the special case of single hidden layer networks, we develop specialized verification algorithms with provable tightness guarantees. 5. We attain state of the art verified bounds on adversarial error rates on image classifiers trained on MNIST and CIFAR-10 under adversarial perturbations in the infinity norm.
 Related Work Certifiable training and verification of neural networks A separate but related thread of work is on certifiable training, ie, training neural networks so that they are guaranteed to satisfy a desired property (for example, robustness to adversarial examples within a certain radius)  [Kolter and Wong, 2018, Raghunathan et al., 2018] . These approaches use ideas from convex optimization and duality to construct bounds on an optimization formulation of verification. However, these approaches are limited to either a class of activation functions (piecewise linear models) or architectures (single hidden layer, as in  Raghunathan et al. [2018] ). Further, in  Kolter and Wong [2018] , the dual problem starts with a constrained convex formulation but is then converted into an unconstrained but nonconvex optimization problem to allow for easy optimization via a backprop-style algorithm. In contrast, our formulation allows for an unconstrained dual convex optimization problem so that for any choice of dual variables, we obtain a valid bound on the adversarial objective and this dual problem can be solved efficiently using subgradient methods. We also note that the ultimate goals of  [Kolter and Wong, 2018, Raghunathan et al., 2018]  are different from our paper: they modify the training procedure of the neural network so that the network is trained to be easily verifiable. In contrast, our work focuses on extending verification algorithms to apply to a broader class of architectures, activation functions and -in this sense, we view our work as complementary to  [Kolter and Wong, 2018, Raghunathan et al., 2018] . In fact, since the objective of our dual optimization is differentiable with respect to the network weights, we can extend our approach to training verifiable networks easily by simultaneously optimizing the network weights and dual variables to minimize the dual objective. We leave the study of such an extension for future work. Theoretical analysis of robustness Another related line of work has to do with theoretical analysis of adversarial examples. It has been shown that feedforward ReLU networks cannot learn to distinguish between points on two concentric spheres without necessarily being vulnerable to adversarial examples within a small radius  [Gilmer et al., 2018] . Under a different set of assumptions, the existence of adversarial examples with high probability is also established in  Fawzi et al. [2018] . In , the authors study robustness of nearest neighbor classifiers to adversarial examples. As opposed to these theoretical analyses, our approach searches computationally for proofs of existence or non-existence of adversarial examples. The approach does not say anything a-priori about the existence of adversarial examples, but can be used to investigate their existence for a given network and compare strategies to guard against adversarial attacks.
 VERIFICATION AS OPTIMIZATION
 NOTATION Our techniques apply to general feedforward architectures and recurrent networks, but we focus on layered architectures for the development in this paper. The input layer is numbered 0, the hidden layers are numbered 1, . . . , L − 1 and the output layer is numbered L. The size of layer l is denoted n l We denote by x in the input to the neural network, by z l the pre-activations of neurons at layer l before application of the activation function and by x l the vector of neural activations after application of the activation function (to z l−1 ). For convenience, we define x 0 = x in . We use x l (x in ), z l (x in ) to denote the activations at the l-th layer as a function of the input x in . Upper and lower bounds on the pre/post activations are denoted by x l , x l , z l , z l respectively. The activation function at layer l is denote h l and is assumed to be applied component-wise, ie, [h l (z l )] k = h l k (z l k ) Note that max-pooling is an exception to this rule -we discuss how max-pooling is handled separately in the Appendix section 6.2.1. The weights of the network at layer l are denoted W l and the bias is denoted b l , z l = W l x l + b l .
 VERIFICATION PROBLEM As mentioned earlier, verification refers to the process of checking that the output of the neural network sat-isfies a certain desirable property for all choices of the input within a certain set. Formally, this can be stated as follows: ∀x in ∈ S in (x nom ) x L (x in ) ∈ S out (1) where x in denotes the input to the network, x nom denotes a nominal input, S in (x nom ) defines the constrained subset of inputs induced by the nominal input, and S out denotes the constraints on the output that we would like to verify are true for all inputs in S in (x nom ). In the case of adversarial perturbations in image classification, x nom would refer to the nominal (unperturbed image), S in (x nom ) would refer to all the images that can be obtained by adding bounded perturbations to x nom , and x in would refer to a perturbed image. In this paper, we will assume that: S out is always a described by a finite set of linear constraints on the values of final layer ie. S out = ∩ m i=1 {x L : c i T x L + d i ≤ 0}, and S in (x nom ) is any bounded set such that any linear optimization problem of the form max x in ∈Sin(x nom ) c T x can be solved efficiently. This includes convex sets and also sets describing combinatorial structures like spanning trees, cuts in a graph and cardinality constraints.
 See the following examples for a concrete illustration of the formulation of the problem: Robustness to targeted adversarial attacks. Consider an adversarial attack that seeks to perturb an input x nom to an input x in subject to a constraint on the perturbation x in − x nom ≤ to change the label from the true label i to a target label j. We can map this to (1) as follows: S in (x nom ) = {x in : x in − x nom ≤ }, (2a) S out = {z : c T z ≤ 0} (2b) where c is a vector with c j = 1, c i = −1 and all other components 0. Thus, S out denotes the set of outputs for which the true label i has a higher logit value than the target label j (implying that the targeted adversarial attack did not succeed). Monotonic predictors. Consider a network with a single real valued output and we are interested in ensuring that the output is monotonically increasing wrt each dimension of the input x in . We can state this as a verifica-tion problem: S in (x nom ) = {x in : x in ≥ x nom } (3a) S out = {x L : x L (x nom ) − x L ≤ 0} (3b) Thus, S out denotes the set of outputs which are large than the network output at x nom . If this is true for each value of x nom , then the network is monotone. Cardinality constraints. In several cases, it makes sense to constrain a perturbation not just in norm but also in terms of the number of dimensions of the input that can be perturbed. We can state this as: S in (x nom ) = {x in : x in − x nom 0 ≤ k, x in − x nom ∞ ≤ } (4a) S out = {z : c T z ≤ 0} (4b) where x 0 denotes the number of non-zero entries in x. Thus, S out denotes the set of outputs which are larger than the network output at x nom . If this is true for each value of x nom , then the network is monotone.
 OPTIMIZATION PROBLEM FOR VERIFICATION Once we have a verification problem formulated in the form (1), we can easily turn the verification procedure into an optimization problem. This is similar to the optimization based search for adversarial examples  [Szegedy et al., 2013]  when the property being verified is adversarial robustness. For brevity, we only consider the case where S out is defined by a single linear constraint c T z + d ≤ 0. If there are multiple constraints, each one can be verified separately. max z 0 ,...,z L−1 x 0 ,...,x L c T x L + d (5a) s.t x l+1 = h l z l , l = 0, 1, . . . , L − 1 (5b) z l = W l x l + b l , l = 0, 1, . . . , L − 1 (5c) x 0 = x in , x in ∈ S in (x nom ) (5d) If the optimal value of this problem is smaller than 0 (for each c, d in the set of linear constraints defining S out ), we have verified the property (1). This is a nonconvex optimization problem and finding the global optimum in general is NP-hard (see Appendix section 6.4.1 for a proof). However, if we can compute upper bounds on the value of the optimization problem and the upper bound is smaller than 0, we have successfully verified the property. In the following section, we describe our main approach for computing bounds on the optimal value of (5).
 BOUNDING THE VALUE OF THE OPTIMIZATION PROBLEM We assume that bounds on the activations z l , x l , l = 0, . . . , L−1 are available. Section 6.1 discusses details of how such bounds may be obtained given the constraints on the input layer S in (x nom ). We can bound the optimal value of (5) using a Lagrangian relaxation of the constraints: max z 0 ,...,z L−1 x 0 ,x 1 ,...,x L−1 c T h L−1 z L−1 + d + L−1 l=0 µ l T z l − W l x l − b l + L−2 l=0 λ l T x l+1 − h l z l (6a) s.t. z l ≤ z l ≤ z l , l = 0, 1, . . . , L − 1 (6b) x l ≤ x l ≤ x l , l = 0, 1, . . . , L − 1 (6c) x 0 ∈ S in (x nom ) (6d) Note that any feasible solution for the original problem (  5 ) is feasible for the above problem, and for any such solution, the terms involving λ, µ become 0 (since the terms multiplying λ, µ are 0 for every feasible solution). Thus, for any choice of λ, µ, the above optimization problem provides a valid upper bound on the optimal value of (5) (this property is known as weak duality  [Vandenberghe and Boyd, 2004] ). We now look at solving the above optimization problem. Since the objective and constraints are separable in the layers, the variables in each layer can be optimized independently. For l = 1, . . . , L − 1, we have f l λ l−1 , µ l = max x l ∈[x l ,x l ] λ l−1 − W l T µ l T x l − b l T µ l which can be solved trivially by setting each component of x l to its upper or lower bound depending on whether the corresponding entry in λ l−1 − W l T µ l is non-negative. Thus, f l λ l−1 , µ l = λ l−1 − W l T µ l + T x l + λ l−1 − W l T µ l − T x l − b l T µ l where [x] + = max(x, 0), [x] − = min(x, 0) denote the positive and negative parts of x. Similarly, collecting the terms involving z l , we have, for l = 0, . . . , L − 1 fl (λ l , µ l ) = max z l ∈[z l ,z l ] µ l T z l − λ l T h l z l where λ L−1 = −c. Since h l is a component-wise nonlinearity, each dimension of z l can be optimized independently. For the k-th dimension, we obtain fl,k λ l k , µ l k = max z l k ∈[z l k ,z l k ] µ l k z l k − λ l k h l k z l k This is a one-dimensional optimization problem and can be solved easily-for common activation functions (ReLU, tanh, sigmoid, maxpool), it can be solved analytically, as discussed in appendix section 6.2. Finally, we need to solve f 0 (µ 0 ) = max x 0 ∈Sin(x nom ) − W 0 T µ 0 T x 0 − b 0 T µ 0 which can also be solved easily given the assumption on S in . We work out some concrete cases in 6.3. Once these problems are solved, we can construct the dual optimization problem: min λ,µ n L−1 k=0 fL−1,k −c k , µ l k + L−2 l=0 n l k=0 fl,k λ l k , µ l k + L−1 l=1 f l (λ l−1 , µ l ) + f 0 µ 0 + d (7) This seeks to choose the values of λ, µ so as to minimize the upper bound on the verification objective, thereby obtaining the tightest bound on the verification objective. This optimization can be solved using a subgradient method on λ, µ. Theorem 1. For any values of λ, µ, the objective of (  7 ) is an upper bound on the optimal value of (5). Hence, the optimal value of (7) is also an upper bound. Further, (7) is a convex optimization problem in (λ, µ). Proof. The upper bound property follows from weak duality  [Vandenberghe and Boyd, 2004] . The fact that (  7 ) is a convex optimization problem can be seen as each term f l , fl,k is expressed as a maximum overa set of linear functions of λ, µ  [Vandenberghe and Boyd, 2004] . Theorem 2. If each h is a ReLU function, then (7) is equivalent to the dual of the LP described in  Ehlers [2017] . Proof. See section 6.4. The LP formulation from  Ehlers [2017]  is also used in  Kolter and Wong [2018] . The dual of the LP is derived in  Kolter and Wong [2018]  -however this dual is different from (7) and ends up with a constrained optimization formulation for the dual (the details of this can be found in appendix section 6.4.2). To allow for an unconstrained formulation, this dual LP is transformed to a backpropagation-like computation. While this allows for folding the verification into training, it also introduces nonconvexity in the verification optimization -our formulation of the dual differs from  Kolter and Wong [2018]  in that we directly solve an unconstrained dual formulation, allowing us to circumvent the need to solve a nonconvex optimization for verification.
 TOWARDS THEORETICAL GUARANTEES FOR VERIFICATION The bounds computed by solving (7) could be loose in general, since (  5 ) is an NP-hard optimization problem (section 6.4.1). SMT solvers and MIP solvers are guaranteed to find the exact optimum for piecewise linear neural networks, however, they may take exponential time to do so. Thus, an open question remains: Are there cases where it is possible to perform exact verification efficiently? If not, can we approximate the verification objective to within a certain factor (known a-priori)? We develop results answering these questions in the following sections. Prior work: For any linear classifier, the scores of each label are a linear function of the inputs w T i x + b i . Thus, the difference between the predictions of two classes j (target class for an adversary) and class i (true label) is (w i − w j ) T x. Maximizing this subject to x − x 0 2 ≤ can be solved analytically to obtain the value (w i − w j ) T x 0 + w i − w j 2 . This observation formed the basis for algorithms in  [Raghunathan et al., 2018]  and  [Hein and Andriushchenko, 2017] . However, once we move to nonlinear classifiers, the situation is not so simple and computing the worst case adversarial example, even in the 2-norm case, becomes a challenging task. In  [Hein and Andriushchenko, 2017] , the special case of kernel methods and single hidden layer classifiers are considered, but the approaches developed are only upper bounds on the verification objective (just like those computed by our dual relaxation approach). Similarly, in  Raghunathan et al. [2018] , a semidefinite programming approach is developed to compute bounds on the verification objective for the special case of adversarial perturbations on the infinity norm. However, none of these approaches come with a-priori guarantees on the quality of the bound, that is, before actually running the verification algorithm, one cannot predict how tight the bound on the verification objective would be. In this section, we develop novel theoretical results that quantify when the verification problem (  5 ) can be solved either exactly or with a-priori approximation guarantees. Our results require strong assumptions and do not immediately apply to most practical situations. However, we believe that they shed some understanding on the conditions under which exact verification can be performed tractably and lead to specialized verification algorithms that merit further study. We assume the following for all results in this section: 1) We study networks with a single hidden layer, i.e. L = 2, with activation function h 0 = h and a linear mapping from the penultimate to the output layer x 2 = h 1 z 1 = z 1 = W 1 x 1 + b 1 . 2) The network has a differentiable activation function h with Lipschitz-continuous derivatives denoted h (tanh, sigmoid, ELU, polynomials satisfy this requirement). 3) S in (x nom ) = {x in : x in − x nom 2 ≤ }. Since the output layer is a linear function of the penultimate layer x 1 , we have c T x 2 = W 1 T c T x 1 + c T b 1 = W 1 T c T h 0 z 0 + c T b 1 For brevity, we simply denote W 1 T c as c, drop the constant term c T b 1 and let W = W 0 , b = b 0 , z nom = W x nom + b and W i denote the i-th row of W . Then, (5) reduces to: max x in : x in −x nom 2 ≤ i c i h i W i x in + b i (8) Theorem 3. Suppose that h has a Lipschitz continuous first derivative: h i (t) − h i ( t) ≤ γ i |t − t| Let ν = diag (c) W T h (z nom ) 2 L = σ max diag (c) W T σ max (diag (γ) W ) Then ∀ ∈ (0, ν 2L ), the iteration: x k+1 ← x nom + W T diag (c) h (W x k + b) W T diag (c) h (W x k + b) 2 starting at x 0 = x nom converges to the global optimum x of (8) at the rate x k − x ≤ L ν− L k Proof. Section 6.4 Thus, when is small enough, a simple algorithm exists to find the global optimum of the verification objective. However, even when is larger, one can obtain a good approximation of the verification objective. In order to do this, consider the following quadratic approxiimation of the objective from (8): max i c i (h i (z nom i ) + h i (z nom i ) (W i z)) + i c i 2 h i (z nom i ) (W i z) 2 (9a) s.t. z 2 ≤ (9b) This optimization problem corresponds to a trust region problem that can be solved to global optimality using semidefinite programming  [Yakubovic, 1971] : max z,Z i c i (h i (z nom i ) + h i (z nom i ) (W i z)) + i c i 2 h i (z nom i ) tr W T i W i Z (10a) s.t. tr (Z) ≤ , 1 z T z Z 0 (10b) where X 0 denotes that X is constrained to be a positive semidefinite matrix. While this can be solved using general semidefinite programming solvers, several special purpose algorithms exist for this trust region problem that can exploit its particular structure for efficient solution,  [Hazan and Koren, 2016]  Theorem 4. Suppose that h is thrice-differentiable with a globally bounded third derivative. Let ζ i = W i 2 , η i = sup t |h i (t)|, κ = 1 6 i η i c i ζ 3 i For each > 0, the difference between the optimal values of (10), (  8 ) is at most κ 3 . Proof. See Section 6.4
 EXPERIMENTS In this section, we present numerical studies validation our approach on three sets of verification tasks: Image classification on MNIST and CIFAR: We use our approach to obtain guaranteed lower bounds on the accuracy of image classifers trained on MNIST and CIFAR-10 under adversarial attack with varying sizes of the perturbation radius. We compare the bounds obtained by our method with prior work (in cases where prior work is applicable) and also with the best attacks found by various approaches. Classifier stability on GitHub data: We train networks on sequences of commits on GitHub over a collection of 10K repositories -the prediction task consists of predicting whether a given repository will reach more than 40 commits within 250 days given data observed until a certain day. Input features consist a value between 0 and 1 indicating the number of days left until the 250th day, as well as another value indicating the progress of commits towards the total of 40. As the features evolve, the prediction of the classifier changes (for example, predictions should become more accurate as we move closer to the 250th day). In this situation, it is desirable that the classifier provides consistent predictions and that the number of times its prediction switches is as small as possible. It is also desirable that this switching frequency cannot be easily be changed by perturbing input features. We use our verification approach combined with dynamic programming to compute a bound on the maximum number of switches in the classifier prediction over time. Digit sum task: We consider a more complex verification task here: Given a pair of MNIST digits, the goal is to bound how much the sum of predictions of a classifier can differ from the true sum of those digits under adversarial perturbation subject to a total budget on the perturbation across the two digits.
 IMAGE CLASSIFICATION: MNIST AND CIFAR We study adversarial attacks subject to an l ∞ bound on the input perturbation. An adversarial example (AE) is a pertubation of an input of the neural network such that the output of the neural network differs from the correct label for that input. An AE is said to be within radius if the ∞ norm of the difference between the AE and the original input is smaller than . We are interested in the adversarial error rate, that is, # Test examples that have an AE within radius Size of test set Computing this quantity precisely requires solving the NP-hard problem (5) for each test example, but we can obtain upper bounds on it using our (and other) verification methods and lower bounds using a fixed attack algorithm (in this paper we use a bound constrained LBFGS algorithm similar to  [Carlini and Wagner, 2017b] ). Since theorem 2 shows that for the special case of piecewise linear neural networks, our approach reduces to the basic LP relaxation from  Ehlers [2017]  (which also is the basis for the algorithms in  Bunel et al. [2017]  and  Kolter and Wong [2018] ), we focus on networks with smooth nonlinearities like tanh and sigmoid. We compare our approach with The SDP formulation from  Raghunathan et al. [2018]  (note that this approach only works for sin-gle hidden layer networks, so we just show it as producing vacuous bounds for other networks). Each approach gets a budget of 300 s per verifiation problem (choice of test example and target label). Since the SDP solver from  [Raghunathan et al., 2018]  only needs to be run once per label pair (and not per test example), its running time is amortized appropriately. Results on smooth activation functions: Figures  1a,1b  show that our approach is able to compute nearly tight bounds (bounds that match the upper bound) for small perturbation radii (up to 2 pixel units) and our bounds significantly outperform those from the SDP approach  [Raghunathan et al., 2018]  ( which is only able to compute nontrivial bounds for the smallest model with 20 hidden units). Results on models trained adversarially: We use the adversarial training approach of  [Uesato et al., 2018]  and train models on MNIST and CIFAR that are robust to perturbations from the LBFGS-style attack on the training set. We then apply our verification algorithm to these robust models and obtain bounds on the adversarial error rate on the test set. These models are all multilayer models, so the SDP approach from  [Raghunathan et al., 2018]  does not apply and we do not plot it here. We simply plot the attack versus the bound from our approach. The results for MNIST are plotted in figure  2b  and for CIFAR in figure  2a . On networks trained using a different procedure, the approaches from Raghunathan et al. [2018] and  Kolter and Wong [2018]  are able to achieve stronger results for larger values of (they work with = .1 in real units, which corresponds to = 26 in pixel units we use here). However, we note that our verification procedure is agnostic to the training procedure, and can be used to obtain fairly tight bounds for any network and any training procedure. In comparison, the results in  Kolter and Wong [2018]  and  Raghunathan et al. [2018]  rely on the training procedure optimizing the verification bound. Since we do not rely on a particular adversarial training procedure, we were also able to obtain the first non-trivial verification bounds on CIFAR-10 (to the best of our knowledge) shown in figure  2a . While the model quality is rather poor, the results indicate that our approach could scale to more complicated models.
 GITHUB CLASSIFIER STABILITY We allow the adversary to modify input features by up to 3% (at each timestep) and our goal is to bound the maximum number of prediction switches induced by each attack over time. We can model this within our verification framework as follows: Given a sequence of input features, we compute the maximum number of switches    [Raghunathan et al., 2018] . Each color represents a different network. The dashed lines at the bottom are lower bounds on the error rate computed using the best attack found using the LBFGS algorithm. achievable by first computing the target classes that are reachable through an adversarial attack at each timestep (using (  7 )), and then running a dynamic program to compute the choices of target classes over time (from within the reachable target classes) to maximize the number of switches over time. Figure  3a  shows how initially predictions are easily attackable (as little information is available to make predictions), and also shows how the gap between our approach and the best attack found using the LBFGS algorithm evolves over time.
 COMPLEX VERIFICATION TASK: DIGIT SUM In order to test our approach on a more complex specification, we study the following task: Given a pair of MNIST digits, we ask the question: Can an attacker perturb each image, subject to a constraint on the total perturbation across both digits, such that the sum of the digits predicted by the classifier differs from the true sum of those digits by as large an amount as possible? Answering this question requires solving the following optimization problem: where s is the true sum of the two digits. Thus, the adversary has to decide on both the perturbation to each digit, as well as the size of the perturbation. We can encode this within our framework (we skip the details here). The upper bound on the maximum error in the predicted sum from the verification and the lower bound on the maximum error computed from an attack for this problem (on an adversarially trained two hidden layer sigmoid network) is plotted in figure  3b . The results show that even on this rather complex verification task, our approach is able to compute tight bounds.
 CONCLUSIONS We have presented a novel framework for verification of neural networks. Our approach extends the applicability of verification algorithms to arbitrary feedforward networks with any architecture and activation function and to more general classes of input constraints than those considered previously (like cardinality constraints). The verification procedure is both efficient (given that it solves an unconstrained convex optimization problem) and practically scalable (given its anytime nature only required gradient like steps). We proved the first known (to the best of our knowledge) theorems showing that under special assumptions, nonlinear neural networks can be verified tractably. Numerical experiments demonstrate the practical performance of our approach on several classes of verification tasks. Figure 1 : 1 Figure 1: Figures show three curves per model -Dashed line: Lower bound from LBFGS attack. Solid line: Our verified upper bound. Dash-dot line: SDP verified upper bound from [Raghunathan et al., 2018] . Each color represents a different network. The dashed lines at the bottom are lower bounds on the error rate computed using the best attack found using the LBFGS algorithm.
 . x in a − x nom a ≤ a , x in b − x nom b ≤ b a + b ≤