Year: 2021

Abstract:
Neural networks have enabled learning over examples that contain thousands of dimensions. However, most of these models are limited to training and evaluating on a finite collection of points and do not consider the hypervolume in which the data resides. Any analysis of the model’s local or global behavior is therefore limited to very expensive or imprecise estimators. We propose to formulate neural networks as a composition of a bijective (flow) network followed by a learnable, separable network. This construction allows for learning (or assessing) over full hypervolumes with precise estimators at tractable computational cost via integration over the input space. We develop the necessary machinery, propose several practical integrals to use during training, and demonstrate their utility.

heading: 1 INTRODUCTION
text: Most supervised learning problems operate by training a model on a finite collection, T , of N (typically paired) examples, (x, y). The model is updated by comparing its predicted output to the expected output and performing some flavor of stochastic gradient descent based on the comparison and various regularizers. The process is repeated by reexamining the elements of T in a random order, possibly with augmentation, until the model parameters converge or an iteration budget is exceeded. This relatively simple procedure has proven to be remarkably effective in a variety of domains and these models have begun to permeate every aspect of modern science and everyday life (Brown et al., 2020; He et al., 2016; Silver et al., 2017).
The deep learning revolution has also resulted in highly effective generative models such as VAEs (Kingma & Welling, 2014), GANs (Goodfellow et al., 2014), and tractable likelihood models (Dinh et al., 2017; Grathwohl et al., 2019). These models are largely used to create novel samples of impressive quality. In addition to sampling, tractable likelihood models also provide an estimate of the probability density function of the data which can be used for additional, downstream processes.
We augment the training process by constructing neural networks that allow for tractable integration over the input domain. This differs from implicit layers which utilize integration over a parameterized variable (Chen et al., 2018; Grathwohl et al., 2019). Access to fast and differentiable integrals allows us to regularize a model’s average behavior using metrics that may not be available otherwise.
Integration over the input space also allows us to supervise how the model behaves in continuous regions that are not directly observed in T and may even be out-of-distribution (OOD). Alternative methods attempt to supervise examples outside of T by performing random perturbations (Gutmann & Hyvärinen, 2010), along the line between known examples (Zhang et al., 2018), or via a generative process (Zenati et al., 2018; Akcay et al., 2018). However, these methods are only capable of observing a small quantity of the total space. By integrating over the entire regions, it is possible to observe a large portion of the space based on statistical relevance.
Main Contributions We propose a general architecture that enables tractable integration over the input space, enabling supervision and custom regularization over continuous regions. We demonstrate how to construct this network and how it allows for a reduction in the computation cost required for dense numeric integration from exponential in the number of dimensions to linear. We derive several useful integrated formulations over continuous regions. Finally, we explore the impact of this architecture and regularizers on the standard accuracy and robustness to OOD examples on several standard classification datasets.
Notation Throughout this work, we consider theM dimensional input features, x = [x1, ..., xM ] ∈ RM ; the latent features, z ∈ Z ⊆ RM ; and the K-wise classification probability, y. The input features x the training set, Tx, are drawn from the in-distribution data, D ⊆ RM . Subscripts represent a particular dimension, e.g., Dm corresponds to the mth dimension of the space. Paranthetical superscripts represent the subspace corresponding to a particular class, e.g., D(c) is the subset of D where the data belongs to class c. The bijective network is given as h such that h : D → Z . Probability distributions overD and Z are given by p with the corresponding subscript. Classification networks are given as f and g. Variables with a “hat,” ŷ, are predictions of the true quantity, y.

heading: 2 MOTIVATION
text: Neural networks are highly effective function approximators between two (typically) continuous spaces: f : X → Y . However, networks are typically trained and evaluated using a finite collection of points without any explicit assessment of the complete hypervolume spanned by the data. This omission is understandable from an implementation perspective as the number of samples required to obtain a reasonable estimate over a volume scales exponentially with data dimensionality. However, human beings often have an understanding of how a process should behave on average. Ideally, we would like to embed this intuition into the model but currently cannot assess the average performance of a trained model outside of the held-out test set. Specifically, we would like to regularize the model by estimating the expected behavior of some metric, Ω, produced by the model over the training data
Ex∼p(x) [Ω(ŷ(x))] = ∫ X Ω(ŷ(x))p(x)dx. (1)
There are many useful choices of Ω over a variety of applications. If it is known what the model output should be on average (ȳ), we can construct Ω to encourage that behavior, e.g., Ω(ŷ) = (ȳ− ŷ)2. Minimizing consistency metrics (Xie et al., 2020) are a common method to improve learning in label-starved problems. These encourage the model to produce similar outputs over neighborhoods around (labeled) examples from Tx where neighborhoods are created by random or domain-specific augmentations. This process can be viewed as an approximation to an integral over the neighborhood,
E ∼p( ) [L(y, ŷ(x + ))] = ∫ L(y, ŷ(x + ))p( )d (2)
where L is a distance-like metric, and is the neighborhood. Equation 2 can be generalized to other neighborhoods. We can recast the standard classification problem as a discrete approximation to an integral. Typically, we minimize the cross-entropy between ŷ(x; θ) and y over the model parameters, θ, for all (x, y) ∈ T which becomes an integral over class-conditioned distributions, p(x|c),
min θ − ∑ x,y∈T ∑ k yk log (ŷk(x; θ))⇒ min θ − ∑ k ∫ D(k) yk log (ŷk(x; θ)) p(x|k)dx. (3)
Unfortunately, integration in high dimension is difficult. Naive gridded solutions require an exponential number of points with error decreasing as O(G−M ), for G, M -dimensional points. Monte Carlo (MC) methods theoretically have better performance with error that decreases asO(G−1/2). However, the rate of convergence for MC methods depends on the variance of samples (Veach, 1998), which may make for poor approximations in practice. Importance sampling (Bishop, 2006) can improve the performance of Monte Carlo methods to adapt to the regions with the greatest contribution.
We choose to model the data using a separable function. Separable functions have the key benefit of decomposing M -dimensional integrals into a combination of M one-dimensional integrals. Each of these one-dimensional integrals can then be solved using any number of highly-accurate solvers (e.g., Runge-Kutta (Shampine, 2005), Dormand-Prince (Dormand & Prince, 1980), Tsit (Tsitouras, 2011), etc.) that have error rates better than O(G−4) but are unavailable in high dimensions. The use of separable functions is a component of the VEGAS algorithm (Peter Lepage, 1978) and is utilized in conjunction with adaptive Monte Carlo sampling to approximate high-dimensional integrals.
The use of a separable function over the input space may make estimation of integrals over the model more accessible; however, they impose a strong, inappropriate, inductive-bias. The obvious solution of using a standard neural network feature extractor and integrating over the learned features means that we would no longer have a valid integrator over the input space. Fortunately, we can solve this
problem by including a bijective transform prior to the separable network as a feature extractor. The bijective transform allows us to decouple the data into a latent space where the data can be modeled using a separable network and guarantees equality between integrals in the latent space and integrals in the input space. The bijective model has the added benefit of allowing us to create a tractable likelihood model (Dinh et al., 2017) over the data which gives us access to p(x) (or possibly p(x|y)) and makes it easier to draw new samples. We discuss bijective and separable functions in Sec. 3, how they come together in Sec. 4, and the specific choices we use in Sec. 5.

heading: 3 BACKGROUND
text: We perform practical integration over the input space by breaking neural network models down into two key components: (1) a bijective feature extractor, (2) a separable task network, see Fig. 1. For the sake of simplicity, we only consider classification tasks in this work. This makes our total network analogous with the common network breakdown where a classifier, often a linear layer or an MLP, is constructed on a feature extractor, such as a CNN. However, unlike the typical process, we must place constraints on both networks so that we can integrate over the input domain. This network breakdown is similar to hybrid networks (Chen et al., 2019; Nalisnick et al., 2019b) except for the separability requirement on the classifier.

heading: 3.1 BIJECTIVE NETWORKS
text: Bijective networks are the key component in flow-based likelihood models. A bijective network, h : D → Z , has a known forward and inverse operation so that data can exactly be reconstructed after the transformation. This allows for exact likelihood estimation via the change of variables formula:
z = h(x; θ), x = h−1(z; θ), pX(x) = pZ(h(x; θ)) ∣∣∣∣∂h∂x ∣∣∣∣ (4)
where pZ is a predefined distribution over the latent space, often a standard Gaussian.
These models are trained via Eq. 4 to maximize the likelihood of the examples in T . Once trained, flow-based likelihood models are commonly used as a generative process where samples are drawn from pZ and are then inverted through h to arrive at an example in D. Our goal is not generative, however it is possible for us to use this process once training is complete. Instead, we will take advantage of the fact that we can choose pZ and then utilize it in downstream tasks. Normalizing flow bijectors provide rich feature extractors that can represent a distribution of complicated inputs with simply-distributed, independent features, z. Given the independence and expressibility of these learned independent features, we build estimators using separable functions over z, which enables us to integrate over the data’s domain while retaining expressibility.
The requirement for bijectivity places a strong constraint on network design, eliminating many common choices due to the need to maintain dimension or invert element-wise activations. Even naive convolutional operations become unavailable since they are not generally invertible. Modern advances have demonstrated methods to work around these limitations through the use of clever partitioning and coupling tricks (Dinh et al., 2017) or the use of constraints (Chen et al., 2019). However, the field of learnable bijective functions is less advanced then its unconstrained counterparts which often results in reduced performance on auxiliary tasks. Flow models are well-studied and We take advantage of several recent architectures to test our methods. In particular, we consider three different bijective models. The simplest models are constructed via stacked affine-coupling layers (Dinh et al., 2017) and invertible, fully-connected layers. We primarily utilize Glow’s (Kingma & Dhariwal, 2018) invertible 1× 1 convolutions to process image data.

heading: 3.2 SEPARABLE FUNCTIONS
text: Separable functions have long been used in mathematics and physics to solve simple partial differential equations such as the homogeneous wave and diffusion equations (Strauss, 2007). We consider two types of separable functions, additive and multiplicative. All proofs can be found in Appendix A.

heading: 3.2.1 ADDITIVELY SEPARABLE FUNCTIONS
text: Definition 3.1. A function, f : CM → CK , is additively separable if it is composed as a summation of element-wise functions operating independently on each dimension of the input:
f (v) = M∑ m=1 fm (vm;φm) (5)
Theorem 3.1 (Additive Independent Integration). Given an additively separable function, f(v), an independent likelihood, p(v) = ∏M m=1 pm(vm), and domain, Dv = Dv1 × ...×DvM :
Ev∼p(v) [f (v)] = M∑ m=1 ∫ Dvm fm (vm) pm (vm) dvm (6)

heading: 3.2.2 MULTIPLICATIVELY SEPARABLE FUNCTIONS
text: Definition 3.2. A function, g : CM → CK , is multiplicatively separable if it is composed as a product of element-wise functions operating independently on each dimension of the input:
g (v) = M∏ m=1 gm (vm;ψm) (7)
Theorem 3.2 (Multiplicative Independent Integration). Given a multiplicitively separable function, g(v), an independent likelihood, p(v) = ∏M m=1 pm(vm), and domain, Dv = Dv1 × ...×DvM :
Ev∼p(v) [g (v)] = M∏ m=1 ∫ Dvm gm (vm) pm (vm) dvm (8)

heading: 4 METHOD
text: Both forms of separable functions allow us to convert a single M -dimensional integral into M 1-dimensional integrals. If we desired to perform a dense estimation of the integral without taking advantage of a separable function using G points per dimension, we would require O(GM ) network evaluations. This is completely impractical for modern datasets where M is at least on the order of hundreds and G should be as large as possible. Exploiting separable functions allow us to reduce the number of network evaluations needed to estimate the integral to O(G). Exploiting separable functions allow us to reduce the computational complexity to estimate the integral to O(G). However, it is unlikely that we could construct a separable function directly on the input space and achieve reasonable performance. Doing so would essentially require that each input dimension contribute to the final output independently of the others. Instead, we can combine Eq. 4 and Eq. 6 (alternatively, Eq. 8) to perform practical integration over the input domain while still allowing for inter-dependent contributions from each input dimension. To do so, we first let the latent distribution, pZ , be independently (but not necessarily identically) distributed: pZ(z) = ∏ m pm(zm). This allows us to write the integral over the input space in terms of the latent space and then simplify via Eq. 6.
Ex∼p(x) [f(h(x))] = Ez∼p(z) [f(z)]
= ∫ D f(h(x))pZ(h(x)) ∣∣∣∣∂h∂x ∣∣∣∣ dx = M∑
m=1
∫ Zm fm(zm) pm (zm) dzm (9)
Each 1-dimensional integral is easily approximated using a variety of integration approaches. In addition to the requirements placed on the feature extractor and task network, this formulation also
requires that we define the integration domain in the latent space as a Cartesian product of domains over each latent dimension. This may seem like a strong requirement since we apparently lose some level of interpretability; however, there are several advantages to defining the input domain in this learned space. Most notably, we know the data distribution in this space exactly and can tune the domain based on the goal of a particular integral. For example, if we are interested in performing average adversarial training, we might choose to integrate in directions that appear less likely as this takes us further from the data domain.
Figure 2 contains a cartoon example that demonstrates how these methods combine to allow for integration over the complex data space. Both plots show the value of f ; Fig. 2a shows the non-separable function in the input space and Fig. 2b shows the same data in the (separable) latent space, after the bijective transformation. The colored points and dotted lines are in correspondence between the two spaces and illustrate how the space is warped by the bijector to create a separable, independent latent space. The light gray lines represent the contours of the underlying probability distribution. We define both the integration domain and perform the integration in the latent space. For simplicity, we illustrate the integration using five, equally-spaced points (in both dimensions). The naive procedure would require sampling f at each point in the grid (the blue points). The use of the separable functions allow us to perform the integral using only the red points (or any combination of the blue points, so long as the value from each row/column is included) and get the same estimate of the integral.

heading: 5 PRACTICAL APPLICATIONS
text: In this section we discuss the choices we make in practice when constructing the latent distribution, separable networks, and the various integrals used. For classification tasks, it is not possible to parameterize the normalized network output of class probabilities, ŷ, as a separable network since each prediction must sum to one. However, it is possible to parameterize each unnormalized logits as a separable network, e.g., ŷ(x) = σ (f(h(x))), where σ is the softmax operator and the output of f are the unnormalized logits. This limits what quantities can be integrated over exactly. Fortunately, it is often possible to integrate over approximations/bounds of a particular metric without resorting to brute-force methods.

heading: 5.1 OUT-OF-DISTRIBUTION DETECTION
text: We construct a global integration regularizaton to provide resilience to out-of-distribution (OOD) examples. We enable OOD detection by introducing a “reject” option (Hendrickx et al., 2021) into the classification vector, increasing the number of classes by one, where the additional class indicates that the instance is OOD. An example is considered OOD if ŷK+1 exceeds a predefined threshold. We supervise the model over out-of-distribution examples by integrating the cross-entropy over the contrastive probability distribution, q(z) (see Sec. 5.4).

heading: 5.2 LATENT DISTRIBUTIONS
text: A critical component of this method is that the likelihoods over the latent space must be independent: pZ(z) = ∏ m pZ(zm). As this is extremely unlikely to occur in the input space, we learn a bijective (flow) transformation from the input space to a latent space where the likelihood can be decomposed into independent components of our choice. The typical choice for the latent distribution is a standard Gaussian. However, since we would like to use the latent features to discriminate between classes
using a separable function, we choose to utilize a (bimodal) Gaussian mixture model. This allows the model to put different classes into one of two “buckets” per feature and enables classification through multiple features. This results in an exponential number of components (with respect to dimension) in the full latent space, e.g., 2M components. However, we can easily offset this by choosing some dimensions to use a unimodal latent distribution while the remaining dimensions are still bimodal
pZm(zm) =
{ 0.5 N (−µ, σ21) + 0.5 N (µ, σ21), if m ≤ L
N (0, σ22), else (10)
In addition to the typical latent data distribution, pZ(z), we also explicitly include a contrastive distribution, q(z). This distribution is critical if we desire to regularize our model outside the data distribution, i.e., setting some background/default behavior. When using a bimodal data distribution, we additionally desire that latent vectors are more likely under this distribution once the vector is sufficiently
far away from any single in-distribution mode. Therefore, we select this distribution so that it fully encapsulates all the components of the data distribution, e.g., qm(zm) > pZm(zm) for | |zm|−µ| > γ. When the data distribution is unimodal we choose pm = qm so that the feature is uninformative.
We utilize a standard Gaussian for the contrastive distribution and set µ to 1 and experiment with σ1 ≤ 0.5. This choice allows us to formulate OOD examples that exist between the in-distribution data as near zero, and outside the in-distribution data, near the tails. Figure 3 illustrates the data and contrastive distributions for a latent feature for convenience.

heading: 5.3 SEPARABLE NETWORKS
text: We explore three different formulations of separable networks. The first formulation learns a distinct quadratic function, the second learns a symmetric hinge, and the third learns a multi-layer perceptron. These functions have distinct parameterizations for each latent feature and each in-distribution class. The out-of-distribution logit is held fixed at zero: lK+1(x) = 0.
f (quad)k,m (zm;αk,m, uk,m, νk,m) = αk,m − (zm − uk,m)2
2e2νk,m (11)
f (hinge)k,m (zm;αk,m, uk,m, νk,m) = αk,m − |zm − uk,m| e νk,m (12)
where k is the logit class index and m is the feature dimension so that the total unnormalized logit, lk(x), is lk(x) = ∑ m fk,m(xm). The separable MLP is constructed by concatenating a learned vector per feature and class to each 1-dimensional feature and constructing a standard MLP that operates on that augmented state. This formulation provides the greatest flexibility as it leverages all the benefits of the universal approximator theorem (Hornik, 1991). Unfortunately, we find the MLP version difficult to train in practice, often resulting in degenerate solutions.
Fixing the OOD logit at zero provides a fixed reference point for the other logits. We can interpret this formulation as each latent feature getting a vote on how in-distribution the example is on a per-class basis. Since each function, fk, is exponentiated after the sum over features (due to the softmax normalization) we can also interpret the process as learning a Gaussian or Laplacian (for the quadratic or hinge parameterization, respectively) over the latent data. This is not exactly true due to the normalization, but it is a convenient intuition.

heading: 5.4 INTEGRALS
text: The cross-entropy, CE(ŷ(x), y), between the normalized model output, ŷ(x), and the true labels, y, is commonly used to train classification models. In this section, we explore global and local integrals of the cross-entropy loss used to train most classifiers (see Eq. 3) when ŷ is given as the composition of the softmax function, a separable function and a bijective function, e.g., σ(f(h(x))). We can express this as minθ ∑ x,y∈T CE(ŷ(x), y) owing to the finite number of elements in T . Ideally, we would minimize the model cross-entropy over all possible examples in D. We accomplish this by defining the subspace, D(c) ⊂ D, that has a given label and corresponding conditional distribution, p(x|c)
Ex∼p(x|c) [CE (ŷ(x), c)] = −Ex∼p(x|c) [log(ŷc(x))] = − ∫ D(c) log(ŷc(x)) p(x|c)dx. (13)
It is not possible to represent ŷ(x) as separable network due to the softmax operation but it is possible to parameterize the unnormalized logits as a separable network. Substituting this parameterization into Eq. 13 and transforming to the latent space yields a bound for the expected cross-entropy
Ex∼p(x|c) [CE (ŷ(x), y)] ≤ − M∑ m=1 ∫ Z(c) fc,m(zm)pm(zm|c)dzm
+ log K+1∑ j=1 M∏ n=1 ∫ Z(c) exp (fj,n(zn)) pm(zm|c)dzn  . (14)
See Appendix E for a full derivation. We will utilize this formulation in conjunction with a contrastive prior to supervise OOD examples, Sec. 5.4.1, and with a local prior to enforce consistency, Sec. 5.4.2.

heading: 5.4.1 OUT-OF-DISTRIBUTION SUPERVISION
text: We can utilize the cross-entropy integral in different ways by choosing the label we would like to apply over a domain. For example, we can integrate the cross-entropy over the OOD latent space, U , with the true label fixed to the reject class (c=K+1), and the latent distribution over codes is the contrastive distribution (Sec. 5.2), p(z|c=K+1) = q(z):
LGLBL = log  K∑ j=1 M∏ n=1 ∫ U exp (fj,n(zn)) qn(zn)dzn  , (15) with fK+1,n=0. Since the data and contrastive distributions overlap, the model could degenerate and always make OOD decisions; however, this would be in conflict with the standard cross-entropy loss applied to each example in the training set. So long as LGLBL is not weighted too strongly, the model achieves equilibrium by making OOD predictions over regions where the contrastive distribution is more likely. In effect, we supervise OOD training without requiring any OOD data.

heading: 5.4.2 LOCAL CONSISTENCY
text: Alternatively, we can perform the integration over neighborhoods of data points in Tx and utilize that point’s label over the entire region. This integral serves to improve consistency around observations. Adversarial attacks (Goodfellow et al., 2015; Madry et al., 2018) are often constructed as perturbations on real data points and adversarial training finds one such point and includes it in the training set. We can interpret local consistency integrations as a form of average adversarial training. We reformulate the integral to be centered around a particular datum, x0 and its class label, y
LLCL = − ∑ k yk ∫ V log(σ (fk(h(x0) + v))) pV (v)dv (16)
A complication resulting from local integration is how to select the local distribution, pV (v), and the neighborhood, V . There are many reasonable choices depending on the goal of the local integration. For simplicity, we choose each Vm ∈ [−ε, ε] and pm(vm) to be uniformly distributed.

heading: 5.5 LOSS COMPONENTS
text: The final training loss used to evaluate the model is composed of different combinations of the standard cross-entropy loss over class predictions, LCE = − ∑ k yk log (ŷ), the negative log-likelihood loss
over in-distribution data points, LNLL = − log(pZ(h(x))) − log ∣∣∂h ∂x ∣∣, and the integration losses, LGLBL and LLCL. We always include LCE and LNLL. Based on results from previous hybrid networks (Chen et al., 2019; Nalisnick et al., 2019b), we weight the negative log-likelihood by 1/M , which is analogous to using bits per dimension and introduce an additional weight over the cross-entropy, λ, which controls the relative importance of the generative and classification tasks. When included, each integration penalty shares the same weight as LCE with additional binary weight, π
Ltotal = 1
M LNLL + λ (LCE + πGLBLLGLBL + πLCLLLCL) . (17)

heading: 6 RELATED WORK
text: Noise Contrastive Distributions Noise contrastive methods introduce a distribution that is distinct from the data distribution, preferably with minimal overlap. The constrastive distribution can be learned and provides a mechanism to supervise the model to discriminate between true and contrastive samples (Gutmann & Hyvärinen, 2010). This forces the model to learn discriminative statistical properties of the data. We utilize this idea to inform the null-space over which we will integrate. We fix the data and contrastive distributions and learn the bijective map between input and latent spaces.
Hybrid Models Normalizing flows are a family of flexible, tractable likelihood estimators based on the application of learnable, bijective functions (Dinh et al., 2017; Grathwohl et al., 2019). These methods have shown strong performance in a variety of domains, both as likelihood estimators and as generative processes. Recent work has coupled these advances in bijective networks to construct invertible feature extractors that are capped by a more conventional classifier. This combination bijective/classifier structure are called hybrid models and show reasonable performance as likelihood and discriminative models (Chen et al., 2019; Nalisnick et al., 2019b). A known shortcoming of these models is that their descriminative performance is lower than those of more traditional models. We utilize a hybrid model structure but with additional constraints over the classification network to enable tractable integration. To our knowledge, other hybrid models construct their classifiers in a way that is less restricted but prohibits practical integration.
Out-of-Distribution Detection Out-of-distribution (OOD) detection is a challenge for many modern machine learning algorithms. An obvious method to assess if an example is OOD would be to train a likelihood model over the data space and threshold since OOD samples should have lower likelihoods than the in-distribution data. Surprisingly, recent work has demonstrated that data that should be OOD actually achieves higher likelihood than the data used in training (Hendrycks et al., 2019; Nalisnick et al., 2019a). Several recent methods have been developed to detect OOD examples including Maximum Softmax Probability (MSP) (Hendrycks & Gimpel, 2017), Outlier Exposure (OE) (Hendrycks et al., 2019), Multiscale Score Matching (MSMA) (Mahmood et al., 2021), ODIN (Liang et al., 2018), and Certified Certain Uncertainty (CCU) (Meinke & Hein, 2020). Lee et al. (2018) utilize a GAN trained with a classifier to produce examples near but out side the training distribution. These methods are constructed specifically for OOD detection whereas our method could be applied to a variety of problems based on the continuous regularizers chosen.
7 EXPERIMENTS
All models were constructed using PyTorch (Paszke et al., 2019), trained using PyTorchLightning (Falcon, 2019), utilized bijectors and distributions from Pyro (Bingham et al., 2018), and were trained using Adam (Kingma & Ba, 2015). Here, we assess the integrable model’s performance using the standard accuracy against a held-out set, the bits-per-dimension (BPD), and the impact on OOD detection. Additional experiments can be found in Appendix B.

heading: 7.1 SPIRALS
text: We construct a synthetic, 2D dataset composed of three intertwined spirals, see Fig. 4a. Suppose that due to an unknown sampling bias, we only observe two of the three arms (missing the green arm) and constructed our model based on these observations as a two-class problem with a third, reject class. See Appendix C for architectures, distributions, and training parameters.
We sample points in a dense 2-dimensional grid at twice the range of the data in both dimensions and evaluate the probability that each point belongs to the two known classes and the OOD class. Figures 4d and 4e illustrate the probability of belonging to the two known classes and Fig. 4f contains the probability that each point is OOD. Red and white values indicate high and low probabilities, respectively. The model successfully identifies the two in-distribution regions. Unsurprisingly, the model also identifies data outside of the training data range as being OOD and, impressively, identifies the unobserved spiral and the region between spirals as being OOD.
Finally, we sample a square box (orange) around a data point (blue) in the latent space and invert the box to assess the appropriateness of the neighborhood in the input space and plot the result, Figs. 4b and 4c. As expected, the bijector converts the latent Euclidean neighborhood into a semantically meaningful neighborhood. Had we included the local cross-entropy integral in this training process, all points within the orange boundary would have received the same label as the data point. However, the fact that the neighborhoods are fairly narrow in directions perpendicular to the manifold indicates that a naive local distribution would have washed out most of the adversarial directions and may reduce the effectiveness of average adversarial training.
7.2 STANDARD PERFORMANCE
We test the impact of integrable models on OOD performance against several standard image classification datasets: MNIST (Deng, 2012), Fashion MNIST (FMNIST) (Xiao et al., 2017), SVHN (Netzer et al., 2011), and CIFAR10 (Krizhevsky et al., 2009). See Appendix B for architures, distributions, and training parameters. Table 1 contains the validation standard accuracy and bits per dimension for all datasets with all integration regularizers. The upper portion of the table is averaged over 10 runs and contains a reject option in the separable model. The lower portion is a single run without a reject option or any integrated regularizers. We find that the model performs reasonably well for MNIST and
FMNIST but the integrated losses cause a large degredation in performance for SVHN and CIFAR10. The removal of these losses produces similar accuracies but much-improved BPD, consistent with the hybrid network results reported by ResidualFlows (Chen et al., 2019) when Glow is used.

heading: 7.3 OUT OF DISTRIBUTION DETECTION
text: We test how our models respond to OOD examples when trained with the global integral over the noise-contrastive prior and the consistency integral and compare to methods designed specifically for OOD detection. See Sec. 6 for a discussion of the baselines. Table 2 contains the AUPR for the various methods against similar but clearly different datasets (the AUROC can be found in Appendix B). We juxtapose SVHN vs CIFAR10 and MNIST vs FMNIST and also include Extended MNIST (EMNIST) (Cohen et al., 2017). We include a separable, hybrid model without integral regularizations (hybrid) and the same model with regularizations (Int. Reg.) to illustrate the utility of learning over hypervolumes. When either MNIST or FMNIST are the in-distribution set, the regularized, integrable network performs on par with the best baseline methods. SVHN and CIFAR10 show reasonable OOD performance but are not as strong as the baselines. This is not surprising since the integrals rely on a reasonable estimate of p(x), which both datasets have failed to achieve, Table 1.

heading: 8 CONCLUSIONS
text: In this work, we develop architectures that enable tractable integration over the data space. We demonstrate that the ability to supervise regions and not isolated points encourages the model to learn better representations and be less likely to degenerate. We consider several formulations that allow us to regularize the model’s behavior based on consistency and contrast without relying on augmentations of and sparse comparisons between a finite collection of data points. We experiment
with the various integrals to obtain promising out-of-distribution detection. Through now tractable integrals, this work enables future methods and applications for learning over continuous regions.

heading: A PROOFS
text: A.1 ADDITIVELY SEPARABLE INTEGRALS FUNCTIONS
Theorem A.1 (Additive Independent Integration). Given an additively separable function, f(v), an independent likelihood, p(v) = ∏M m=1 pm(vm), and domain, Dv = Dv1 × ...×DvM :
Ev∼p(v) [f (v)] = M∑ m=1 ∫ Dvm fm (vm) pm (vm) dvm (A1)
Proof. ∫ fn (vn) pm (vm) dvm = { fn (vn) , if n 6= m∫ fn (vn) pn (vn) dvn, else
(A2)
Ev∼p(v) [f (v)] = ∫ D ( M∑ n=1 fn (vn) )( M∏ m=1 pm(vm) ) dv (A3)
= ∑ n ∫ D fn (vn) M∏ m=1 pm(vm) dv (A4)
Substituting Eq. A1 into Eq. A3 reduces the M -dimensional integral over independent likelihoods into a 1-dimensional integral and completes the proof.
A.2 MULTIPLICATIVELY SEPARABLE INTEGRALS FUNCTIONS
Theorem A.2 (Multiplicative Independent Integration). Given a multiplicitively separable function, g(v), an independent likelihood, p(v) = ∏M m=1 pm(vm), and domain, Dv = Dv1 × ...×DvM :
Ev∼p(v) [g (v)] = M∏ m=1 ∫ Dvm gm (vm) pm (vm) dvm (A5)
Proof. Since each component of gn is matched to a component in pm, the two products can be recombined to isolate like-dimensions. Each dimension can then be integrated independently.
Ev∼p(v) [g (v)] = ∫ dv ( M∏ n=1 gn(vn) )( M∏ m=1 pm(vm) )
= ∫ · · · ∫ ( M∏
n=1
gn(vn)pn(vn) ) dv1 · · · dvM
= ∫ g1(v1)p1(v1)dv1 · · · ∫ gM (vM )pM (vM )dvM

heading: B ADDITIONAL EXPERIMENTS
text: All image datasets were trained using Glow as the bijector with the number of levels adjusted for the size of the image. MNIST and Fashion MNIST were trained with Adam for 50 epochs with an exponentially decaying learning rate of γ = 0.75. SVHN and CIFAR10 utilized a similar training procedure but with 75 epochs and a slower exponential decay of γ = 0.99.
B.1 ADVERSARIAL ROBUSTNESS
We test the efficacy of the local cross-entropy integral by utilizing a synthetic dataset from (Tsipras et al., 2019). The data is constructed via
y ∼ {−1, +1}, x1 = {
+y, w.p. p −y, w.p. 1− p , x2, ..., xM+1 ∼ N (ηy, 1) (B1)
and the adversarial examples are constructed analytically by, essentially, flipping the sign of η. We utilize this synthetic dataset to test adversarial robustness because it provides an upper bound on the adversarial accuracy relative to the standard accuracy which allows us to quantify our performance with respect to an absolute. Additionally, because adversarial examples can be constructed exactly without relying on an optimization procedures, there can be no concerns that our defense relies on obfuscated gradients (Athalye et al., 2018). We refer the interested reader to the original paper for a thorough discussion.
We choose M = 10, p = 0.95, and η = 2/ √ M . For these choices, a strong classifier will obtain near perfect standard accuracy with zero adversarial accuracy while a robust classifier will obtain 95% standard and adversarial accuracy. We use a similar architecture to that found in Sec. C.1 except we utilize ten blocks instead of five. We train one model with only the standard crossentropy and negative log-likelihood losses and a second model with the additional global and local cross-entropy integration loss. We find that the model with only the standard losses achieves reasonably good standard performance and fairly poor adversarial
accuracy. The model with the additional integration losses contains considerably better adversarial performance, nearing the upper and, necessarily lower, standard accuracy bound. Table 3 summarizes our results and the ideal performances.
B.2 TOY SEMI-SUPERVISED REGRESSION
To illustrate the utility of global regularizations, we construct a one-dimensional, semi-supervised problem with x ∼ N (0, 4) and y = tanh(x). We keep all values of x but remove y values corresponding to negative x during training. Unlike the standard semi-
supervised problem, the missing labels are not random but are the result of limitations or bias in the data collection process. We train two models that are identical except that the integrated model includes a penalty over E [ŷ (x)]. Table 4 shows how the models perform over the supervised and unsupervised regions. The inclusion of the integration regularizer allows the model to reason about regions that it has not directly observed and has decreased the error in that region by two orders of magnitude.
B.3 OUT OF DISTRIBUTION DETECTION AUROC COMPARISONS
B.4 INTERPRETABILITY
The separably model and independent latent dimensions allows us to reason about how the model is making decisions in the latent space. Unfortunately, for most bijectors, it is not possible to carry this interpretability back to the input space. Figure 5 demonstrates the final state of the model trained on Fashion MNIST for several features with respect to the latent distribution, per in-distribution class and juxtaposed with the out of distribution data. Specifically, the top row contains the logit components, fk,m, learned by the separable network, color-coded by class; the middle row contains the distribution of each class (colors matched to logits); and the bottom row contains the distribution
of the in-distribution data (blue) and OOD data (red). The top row illustrates how the classification network makes its decisions per feature over the distribution presented in the middle row. We see that the logit components map well to the distribution of the data in each feature and provides some intuition for how certain the classifier is over a feature. This demonstrates how this architecture allows for reasonable interpretibility from the latent space. Any value above zero (the dotted black line) is considered in-distribution and the most likely class is the line with the greatest value at that point. The features were chosen to demonstrate the diversity of the learned solution over features. Generally, the data maps reasonably well to the bimodal distribution, though we do occassionally see mode collapse as in Fig. 5e. Fortunately, in these cases the logits tend to be fairly uninformative and only introduce a small bias. Figures 5a through 5c show a common trend where the OOD data has heavy overlap with one of the two clusters but not the other. While we do see some diversity amongst the in-distribution classes that overlap with the OOD data the “bags” (gray) and “sandals” (cyan) class overlap most often. Finally, Fig. 5d demonstrates a latent feature where the OOD data falls in the region between the two data components.
B.5 MNIST LEAVE-ONE-OUT
Following (Ahmed & Courville, 2020), we test the robustness of our method against semantic OOD examples by training models on all but one class from MNIST (Deng, 2012) and assessing OOD performance on the held out class. We repeat the process for each class and report the AUROC and AUPR. In all cases, we train using ten different random seeds and report the average performance and standard deviation across seeds that do not degenerate against the in-distribution validation set. Table 7 and Table 6 contain the results of our method using both integrated losses compared to several relevant baselines. Results from Efficient GAN-Based Anomaly Detection (EGBAD) (Zenati et al., 2018) and GANomaly (Akcay et al., 2018) are taken by estimating the results from figures in both works. The Semantic Anomalies (Ahmed & Courville, 2020) results are obtained by executing the official version of the code using the rotation auxiliary task with two different batch sizes (128 and 256). The Semantic Anomalies results in both tables are the best across both batch sizes and detection methods (ODIN (Liang et al., 2018) and MSP (Hendrycks & Gimpel, 2017)) based on the AUPR.
We see that the Semantic Anomalies generally achieves the best AUROC across all digits. The integration often achieves the best AUPR but struggles with certain held out digits. In particular, the integration performs significantly worse on the “4” and “9” digits. This is a reasonable confuser due to the similarities and variability of the two classes. These results indicate that the integration penalties are helpful for inducing the model to detect semantic anomalies.

heading: C TRAINING AND ARCHITECTURES
text: The bijective layer is composed of five blocks where each block contains an affine-coupling layer (Dinh et al., 2017) and an invertible fully connected layer. The data distribution is bimodal over both latent dimensions with means at ±1 and standard deviations of 0.4. The separable network is composed of quadratic functions. We train the model as discussed, using the standard cross-entropy loss for each observed point, the negative log-likelihood over the input space, and the global crossentropy integration with respect to the contrastive prior. The learning rate is set to 0.001 with standard weight decay of 1e-4 over 20 epochs using Adam (Kingma & Ba, 2015).
C.2 FASHION MNIST
The bijective layers are composed of the Glow (Kingma & Dhariwal, 2018) architecture with two levels composed of 32 steps and 512 channels. We utilize an open-source Glow implementation1 wrapped in Pyro’s (Bingham et al., 2018) bijective operators. The data distribution is bimodal over all latent dimensions with means at±1 and standard deviations of 0.5. The noise constrastive distribution is a standard Gaussian. The separable network is composed of hinge functions (see Sec. 4.3). We utilize a batch size of 256, a learning rate of 1.5e-4 over the bijective layers, and a learning rate of 1.0e-3 over the separable layers using Adam (Kingma & Ba, 2015). Both learning rates are
1https://github.com/chrischute/glow
exponentially decayed at a rate of 0.75 per epoch. Standard weight decay is applied with a weight of 1e-4. The network is trained for 50 epochs. The adversarial attacks are performed against the model at the epoch with the lowest validation cross-entropy loss.

heading: D APPROXIMATE EXPECTED CROSS-ENTROPY OVER A DOMAIN
text: We desire the expected cross-entropy, CE(ŷ(x), yc), over a domain Dc where yc is the desired probability vector over classes, ŷ(x) is the model’s prediction of y ∈ RK from x ∈ RM and is composed of a bijective function, h, a separable function, fk(x) = ∑ m fk,m(xm), and the soft-max function, σ, such that ŷ(x) = σ(f(h(x))). If we are attempting to regularize the model to produce a single class label over Dc, then y will correspond to a one-hot vector. In general, however, yc may be a dense vector with elements yk. The expected cross-entropy is expressed as
EDc [CE(ŷ(x), yc)] = − K∑ k=1 yk EDc [log(ŷk(x))] = − K∑ k=1 yk ∫ Dc log(ŷk(x)) pDc(x|y)dx (D1)
where pDc(x|y) is the probability density function over Dc conditioned on y. We can take advantage of the bijective transform to convert this expression into an expectation over the latent space, z, and latent domain, Zc, which allows us to write
K∑ k=1 yk EDc [log(ŷk(x))] = K∑ k=1 yk EZc [log(σ(f(z))] = K∑ k=1 yk ∫ Zc log(σk(f(z))) pZc(z|y)dz
(D2)
where σk is the kth element after the soft-max normalization and pZc(z|y) is the independent probability density function in the latent space, e.g., pZc(z|y) = ∏ m pm(zm). We can then expand the soft-max operator within the expectation to get
K∑ k=1 yk EZc [log(σ(f(z))] = −EZc log  K∑ j=1 exp(fj(z)) + K∑ k=1 yk EZc [fk(z)] (D3)
The combination of the separable function and independent densities allows for an exact simplification of the second term into
K∑ k=1 yk EZc [fk(z)] = K∑ k=1 yk M∑ m=1 ∫ Zm fk,m(zm)pm(zm)dzm (D4)
≈ 1 G K∑ k=1 yk M∑ m=1 G∑ g=1 fk,m(z̃m,g)
where we have overloaded Zm to correspond to the domain of the mth latent dimension and have approximated the integral via Monte Carlo integration with z̃m,gp̃m(zm) and G draws. The first term on the right-hand side of Eq. D3 does not enjoy the same exact simplification. However, it is possible to bound the term via Jensen’s Inequality by moving the logarithm into the sum over j or outside the expectation operator. We consider the latter case where
EZc log  K∑ j=1 exp(fj(z))  ≤ log EZc  K∑ j=1 exp(fj(z))  . (D5) Then, we expand the separable function, exchange the order of the sum over dimensions and the exponent, take advantage of the multiplicative integral simplification (Eq. A5), approximate via
Monte Carlo integration, and utilize the log ∑ exp operator for stability:
log EZc  K∑ j=1 exp(fj(z))  = log EZc  K∑ j=1 exp( M∑ m=1 fj,m(zm))  (D6) = log
 K∑ j=1 ∫ Zc dz M∏ m=1 exp(fj,m(zm)) M∏ n=1 pn(zn)  = log
 K∑ j=1 M∏ m=1 ∫ Zm exp(fj,m(zm))pm(zm)dzm  ≈ log
 K∑ j=1 M∏ m=1 G∑ g=1 exp(fj,m(z̃m,g)) −M log(G) = log
K∑ j=1 exp M∑ m=1 log G∑ g=1 exp(fj,m(z̃m,g))−M log(G).
Finally, we can substitute Eq. D4 and Eq. D6 into Eq. D1 to get
EDc [CE(ŷ(x), yc)] ≤ − K∑ k=1 yk M∑ m=1 ∫ Zm fk,m(zm)pm(zm)dzm (D7)
+ log  K∑ j=1 M∏ m=1 ∫ Zm exp(fj,m(zm))pm(zm)dzm  ≈ − 1
G K∑ k=1 yk M∑ m=1 G∑ g=1 fk,m(z̃m,g)
+ log K∑ j=1 exp M∑ m=1 log G∑ g=1 exp(fj,m(z̃m,g))−M log(G)
In the event that yc is a one-hot vector (at the cth element) this simplifies to
EDc [CE(ŷ(x), yc)] ≤ − M∑ m=1 ∫ Zm fc,m(zm)pm(zm|c)dzm (D8)
+ log  K∑ j=1 M∏ m=1 ∫ Zm exp(fj,m(zm))pm(zm|c)dzm  where we reintroduced the condition over the class within the probability densities pm(zm|c) for emphasis.

Review summaries:
This paper introduces a hybrid model architecture that makes it possible to integrate a separable loss function across a region of input space. Such integrals can be used as regularizers for robustness near the observed data points (local consistency), and out-of-distribution (OOD) detection in neighborhoods away from the observations. The paper uses these regularizers to train models that are less vulnerable to adversarial attacks and out of distribution mishaps.

For input-output pair $(x,y)$, the paper undertakes the task of estimating (*) $E_{x \sim p(x)} \Omega(\hat y(x))$. When $x$ is high dimensional, integration is difficult. Standard ML/DL applies MC to estimate this integral based on an iid sample $(x_i,y_i), i=1,\ldots,n$. This paper suggests there is a better way.   The main idea is to recognize that if $f$ is a separable function and $h$ is a bijective function, then (**) $E_{x \sim p(x)} f(h(x))$ is easy to evaluate.