Practical Integration via Separable Bijective Networks
Authors: Christopher M Bender
Patrick Emmanuel
Michael K. Reiter
Junier Oliva
Venue: ICLR 2022 Poster (2022)
Keywords: integration
flow
likelihood
classification
regression
out of distribution
regularization

Abstract: Neural networks have enabled learning over examples that contain thousands of dimensions.
However, most of these models are limited to training and evaluating on a finite collection of \textit{points} and do not consider the hypervolume in which the data resides.
Any analysis of the model's local or global behavior is therefore limited to very expensive or imprecise estimators.
We propose to formulate neural networks as a composition of a bijective (flow) network followed by a learnable, separable network.
This construction allows for learning (or assessing) over full hypervolumes with precise estimators at tractable computational cost via integration over the \textit{input space}.
We develop the necessary machinery, propose several practical integrals to use during training, and demonstrate their utility.

Published as a conference paper at ICLR 2022
PRACTICAL INTEGRATION VIA SEPARABLE BIJECTIVE NETWORKS
Christopher M. Bender'2, Patrick R. Emmanuef" Michael K. Reiter', Junier B. Olival 'Department of Computer Science, The University of North Carolina "hh Johns Hopkins University Applied Physics Laboratory DDeparteent of Computer Science, Duke University (bender, joliva/ (cs. unc. unc.edu, patrick. wtw.ricc.eemmmueee jhuapl edu, michael rei weiterte tereeuke edu
ABSTRACT
Neural networks have enabled learning over examples that contain thousands dimensions. However, most of these models are limited to training and evaluating on a finite collection of points and do not consider the hypervolume in which the data resides. Any analysis of the model's local global behavior is therefore limited to very expensive or imprecise estimators. We propose to formulate neural networks composition of a bijective (flow) network followed by learnable, separable network. This construction allows for learning (or assessing) over full hypervolumes with precise estimators at tractable computational cost via integration over the input space. We develop the necessary machinery, propose several practical integrals to use during training, and demonstrate their utility.
INTRODUCTION
Most supervised learning problems operate by training a model on finite collection, T, of N (typically paired) examples, (Ir, y). The model is updated by comparing its predicted output to the expected output and performing some flavor of stochastic gradient descent based the comparison and various regularizers. The process is repeated by reexamining the elements of T in a random order, possibly with augmentation, until the model parameters converge an iteration budget is exceeded. This relatively simple procedure has proven to be remarkably effective in a variety of domains and these models have begun to permeate every aspect of modern science and everyday life (He et al. 2016 Silver et al. 2017 Brown et al. 2020).
The deep learning revolution has also resulted in highly effective generative models such VAEs (Kingma & Welling 2014). GANs (Goodfellow et al. 2014)) and tractable likelihood models (Dinh aall||20: 2017] Oliva et al.) 20188; Grathwohl et al. (2019). These models are largely used to create novel samples of impressive quality. In addition to sampling, likelihood models provide an estimate of the probability density function of the data which can be used for additional, downstream processes.
We augment the training process by constructing neural networks that allow for tractable integration over the imput domain. This differs from implicit layers which utilize integration over a parameterized variable (Chen et al. 2018 Grathwohl et al. 2019). Access to fast and differentiable integrals allows us to regularize model's 's average behavior using metrics that may not available otherwise. Integration over the input space also allows us supervise how the model behaves in continuous regions that are not directly observed in T and may even be out-of-distribution (OOD).
Alternative methods attempt to supervise examples outside of T by performing random perturbations (Guimann & Hyvärinen] 2010), along the line between known examples (Zhang et al., 2018), or via a generative process (Zenati al.) 2018 [Akea al., 2018). However, these methods only capable of observing a small quantity of the total space. By integrating over entire regions, it is possible to observe a large portion of the space based statistical relevance.

---

Published as a conference paper at ICLR 2022
Main Contributions We propose a general architecture that enables tractable integration over the input space, enabling supervision and custom regularization over continuous regions. We demonstrate how to construct this network and how it allows for reduction in the computation cost required for dense numeric integration from exponential in the number of dimensions to linear. We derive several useful integrated formulations over continuous regions. Finally, the impact of this architecture and regularizers on the standard accuracy and robustness to OOD examples on several standard classification datasets. The code utilized in this paper can be found at https: / //ggtttuuuu github........ com/ Lupalab/ sep bi j_ntts.
explore
Notation Throughout this work, we consider the M dimensional input features, x [a1,,-., Tar] E RM; the latent features, z E Z S RM; and the K-wise classification probability, y. The input features x the training set, Tz, are drawn from the in-distribution data, D C IR Subscripts represent particular dimension, c.g., Dm corresponds to the m²t dimension of the space. Paranthetical superscripts represent the subspace corresponding to particular class, e.g., D(e) is the subset of 'D where the data belongs to class c. The bijective network is given as l such that h D + Z. Probability distributions over DD and 2 are given by p with the corresponding subscript. Classification networks are given as f and Variables with a "hat," G, are predictions of the true quantity, y.
2 MOTIVATION
Neural networks are highly effective function approximators between two (typically) continuous spaces: f X' V. However, networks are typically trained and evaluated using a finite collection of points without any explicit assessment of the complete hypervolume spanned by the data. This omission is understandable from an implementation perspective as the number of samples required to obtain a reasonable estimate over volume scales exponentially with data dimensionality. However, human beings often have an understanding of how a process should behave on average. Ideally, we would like to embed this intuition into the model but currently cannot assess the average performance of trained model outside of the held-out test set. Specifically, we would like to regularize the model by estimating the expected behavior of some metric, f2 produced by the model over the training data
/ ((x(())p(xxxxx
Ex-p(x) [[(((())]
(I)
There are many useful choices of Ω over a variety of applications. If it is known what the model
output should be on average (j), we can construct 02 to encourage that behavior, e.g., Q()) = (ū - -
Minimizing consistency metrics (Xie et al., 2020) are common method to improve learning in label-starved problems. These encourage the model to produce similar outputs over neighborhoods around (labeled) examples from Tz where neighborhoods are created by random or domain-speciffe augmentations. This process can be viewed as an approximation to an integral over the neighborhood,
Eecple) [C(y, 0(x + +))]] / Cly, 0(x + e))p(e)de
where C is a distance-like metric, and is the neighborhood. Equation 2 can be generalized to other neighborhoods. We can recast the standard classification problem as a discrete approximation to an integral. Typically, we minimize the cross-entropy between (xx; #) and y over the model parameters, 0. for all (x, v) E T which becomes an integral over conditioned distributions, p(x(c).
min-
Σ> &
Σ
yk log (1; 0)) min 0
Σ
Dc: log (Ua(x; 0)) p(x[k)dxx
(3)
Z.BET
Unfortunately, integration in high dimension is difficult. Naive gridded solutions require an exponen- tial number of points with error decreasing as ( M) for G, -dimensiional points. Monte Carlo (MC) methods theoretically have better performance with error that decreases as O(G-1/2) However, the rate of convergence for MC methods depends the variance of samples (Veach 1998), which may make for poor approximations in practice. Importance sampling (Bishop) (006)) can improve the performance of Monte Carlo methods to adapt to the regions with the greatest contribution.
We choose to model the data using a separable function. Separable functions have the key benefit of decomposing M-dimensional integrals into combination of M one-dimensional integrals. Each
2

---

Published as a conference paper at ICLR 2022
Bijective
Separable Network
X
Network
Figure 1: Depiction of the overall network with intervening distributions over the latent space, Z
of these din-eimnniional integrals can then be solved using any number of highly-accurate solvers (e.g.. Sounge-Kutten (Shampine] 2005). Dormand- Prinne (Dormand & Prince 1980), Tsit (Tsitouras 2011), etc.) that have error rates better than O 4) but unavailable in high dimensions. The of separable functions a component of the VEGAS algorithm (Peter Lepage| 1978) and is utilized in conjunction with adaptive Monte Carlo sampling to approximate high-dimensional integrals.
The use of a separable function over the input space may make estimation of integrals over the model more accessible; however, they impose strong, inappropriate inductive-bias. The obvious approach of utilizing a standard neural network as feature extractor and integrating over learned features means that we would no longer have valid integrator over the input space. We propose solve this problem by utilizing bijective transforms prior to the separable network. The bijective transform allows us to decouple the data into a latent space where the data can modeled using separable network and guarantees equality between integrals in the latent space and integrals in the input space.
3 BACKGROUND
We perform integration over the input space by splitting neural network models down into two key components: (1) a bijective feature extractor, (2) separable task network, see Fig. T For simplicity, we only consider classification tasks in this work. This makes our total network analogous with the common architecture where classifier, often linear layer or an MLP, is constructed on feature extractor, such as a CNN. Unlike the typical process, we must place constraints on both networks so that we can integrate over the input domain. This network breakdown similar to hybrid networks (Chen et al. 2019 Nalisnick et al. 2019b) except for the separability requirement on the classifier.
3.1 BLEECIVE NETWORKS
Bijective networks are the key component in flow-based likelihood models. A bijective network, h: D + Z. has known forward and inverse operation that data exactly be reconstructed after the transformation. This allows for exact likelihood estimation via the change of variables formula:
Oh
h(x: 0), - -z; 0), (x pz pz (h(x: 0))
Ox
where pz a predefined distribution over the latent space, often a standard Gaussian.
These models are trained via Eq. to maximize the likelihood of the examples in T. Once trained, flow-based likelihood models are commonly used as generative process where samples are drawn from pz and are then inverted through / arrive an example in D. Instead, will take advantage of the fact that we can choose pz and then utilize it in downstream tasks. Normalizing flow bijectors provide rich feature extractors that can represent distribution of complicated inputs with simply- distributed, independent features, 2. Given the independence and expressibility of these learned independent features, we build estimators using separable functions over 2, which enables us to integrate over the data's domain while retaining expressibility.
The requirement for bijectivity places a strong constraint on network design, eliminating many common choices due to the need to maintain dimension invert element-wise activations. Even naive convolutional operations become unavailable since they are not generally invertible. Modern advances have demonstrated methods to work around these limitations through the use of clever partitioning and coupling tricks Dinh et al. (2017) or the use constraints Chen et al. (2019). However, the field of learnable bijective functions less advanced then its unconstrained counterparts which results in reduced performance on auxiliary tasks. We utilize Glow Kingma & Dhariwal (2018) to process image data and continuous normalizing flows Grathwohl et al. (2019) for tabular data.
3

---

Published as a conference paper at ICLR 2022
3.2
SEPARABLE FUNCTIONS
Separable functions have long been used in mathematics and physics to solve simple partial differential equations such the homogeneous types of separable functions, additive and multiplicative. All proofs can be found in Appendix A.
and diffusion equations (Strauss 2007). We consider two
3.2.1 ADDITIVELY SEPARABLE FUNCTIONS
Definition 3.1. function, f C. + c is additively separable if is composed as a summation of element-wise functions operating independently each dimension of the input:
f (v)
E fm (Umi Pm)
(5)
Theorem 3.1 (Additive Independent Integration). Given an additively separable function, f(v), an independent likelihood, p(v) = II Pm (Fm), and domain, D, = Dvy X X Duss
M Ev-p(v) [f) Σ
[ fm (Um) Pm (Um) dum
m=
Dum
3.2.2 MULTIPLICATIVELI SEPARABLE FUNCTIONS
Definition 3.2. A function, 9 C. cK is multiplicatively separable if it is composed product element-wise functions operating independently on each dimension of the input:
M
(v)
II m=1
Im (Umi mm)
(7)
Theorem 3.2 (Multiplicative Independent Integration). Given a multiplicitively separable function, g(v), an independent likelihood, p(v) = II Pm (Em), and domain, D, = Dv, X X Duxi
M II
/ Dum
Evpp(y) [v))
Im (Um) Pmn (Um) dum.
(8)
4 METHOD
Both forms of separable functions allow us to decompose a single M -ddimensionnl integral into M 1-dimensional integrals. A dense estimation the integral without taking advantage of separable function using G points per dimension would require ((G*) network evaluations. This is completely impractical for modern datasets where M is least on the order of hundreds and G should be as large as possible. Exploiting separable functions allow us to reduce the complexity to O(GM).
However, it is unlikely that we could construct separable function directly on the input space and achieve reasonable performance. Doing would essentially require that each input dimension contribute the final output independently of the others. Instead, we can combine Eq. 4 and Eq. 6 (alternatively, Eq. 8) to perform practical integration over the input domain while still allowing for inter-dependent contributions from each input dimension. To do so, we first let the latent distribution, Pz. be independently (but not necessarily identically) distributed: (2(2) IIn,, (2..) This allows us to write the integral over the input space in terms of the latent space and then simplify via Eq. 6
J f(b(x))p2(((x))
Oh
Exxpx [f(()))
0x
M Σ m=1
[f(z)] = / Pz p2(2)s(z)dz
E (x)
S (Zm) Pm (Zm) däm
2...
Each I-dimensional integral can be easily approximated using a variety of integration approaches. In addition to the requirements placed the feature extractor and task network, this formulation also
4

---

Published as a conference paper at ICLR 2022
requires that we define the integration domain in the latent space Cartesian product of domains over each latent dimension. This may seem like a strong requirement since we apparently lose some level of interpretability; however, there several advantages defining the input domain in this learned space. Most notably, we know the data distribution this space exactly and can tune the domain based the goal of partieular integral.
Figure contains a cartoon example that demonstrates how these methods combine allow for integration over the complex data space. Both plots show the value of f: Fig. 2a shows the non- separable function in the input space and Fig. 2b shows the same data in the (separable) latent space, after the bijective transformation. The colored points and dotted lines are in correspondence between the two spaces and illustrate how the space is warped by the bijector to create a separable, independent latent space. The light gray lines represent the contours of the underlying probability distribution. We define both the integration domain and perform the integration in the latent space. For simplicity, we illustrate the integration using five, squally-spacee points (in both dimensions). The naive procedure would require sampling
Sep, Samples
(a) Input Space f (h(())
each point in the grid (the blue points). The use of the separable functions allow us to integrate using only the red points and get the same estimate.
PRACTICAL APPLICATIONS
In this section we discuss specific choices made when constructing the latent distribution, separable networks, and various integrals. For classification tasks, it is not possible to parameterize the normalized network output of class probabilities, ÿ. as separable network since each prediction must sum to one. However, it is possible to parameterize each unnormalized logits as a separable network, e.g., (xx) a (f(hx)))), where is the softmax operator and the output of f are the unnormalized logits. While this limits what quantities can be integrated over exactly, it is still possible to integrate over approximations/bound, without resorting to brute-force methods.
(b) Latent Space f(z)
Figure 2: Separable func- tions need O latent sam- ples (red) instead of ((GM) input samples (blue). Integra- tion regions emphasized.
ontttf-Disttibution Detection We construct a global integration regularizer to provide resilience to out-of-distribution (OOD) examples. We enable OOD detection by introducing a "reject" option (Hendrickx et al., 2021) into the classification vector, increasing the number of classes by one, where the additional class indicates that the instance is OOD. An example is considered OOD if (K+1 exceeds a predefined threshold. We supervise the model over out-of-distribune integrating the cross-entropy over the contrastive probability distribution, (z)) (see Sec. 5.3.
examples by
Semi-supervised Learning We build local integral to enhance consistency within latent neighbor- hoods and utilize it in place of pre-chosen augmentation strategies to inform a label-starved dataset. Specifically, we introduce loss where all examples
real, labeled example are regularized to share the real example's label as in Eq. 2 (see Sec. 5.3.2). We additionally use pseudo-labele [ee] (2013) so that consistency maintained about unlabelled points the model grows more confident.
5.1 LATENT DISTRIBUTIONS
critical component of this method is that the likelihoods over the latent space must be independent: (z( = IIm m Pz(zm)) As this is extremely unlikely to occur in the input space, we learn a bijective (flow) transformation from the input space to latent space where the likelihood can be decomposed into independent components of our choice. Since we would like use the latent features discriminate between classes using a separable function, we choose to utilize a (bimodal) Gaussian mixture model. This allows the model to put
Exroppe Deenibbttoss
Figure 3: Data and con- trastive distributions
5

---

Published as a conference paper at ICLR 2022
different classes into one of two "buckets" per feature and enables classification through multiple features. This results in an exponential number of components (with respect to dimension) in the full easily offset this by choosing some dimensions
latent space, e.g., 2M components. However, to use a unimodal latent distribution while the remaining dimensions are still bimodal:
[0.5.V(-H,.)) of) + (H, of, if o0)
Pzm (Zm)
(10)
else
In addition to the typical latent data distribution, pz (z), we explicitly include a contrastive distribution, qz)). This distribution is critical if desire regularize our model outside the data distribution, i.e., setting some background/default behavior. When using a bimodal data distribution, we additionally desire that latent vectors are more likely under this distribution once the vector is sufficiently far away from any single in-distribution mode. Therefore, we select this distribution so that it fully encapsulates all the components of the data distribution, e.g., qm (2m)) (2m) pzm (2m) for | - > 7. When the data distribution is unimodal we choose pm 4m so that the feature is uninformative.
We utilize a standard Gaussian for the contrastive distribution and set μ to and experiment with 01 S 0.5. This choice allows us to formulate OOD examples that exist between the in in-distribution data as near zero, and outside the in-distribution data, near the tails. Figure Jillustratess the data and contrastive distributions for latent feature for convenience.
5.2 SEPARABLE NETWORKS
We explore three different formulations of separable networks. The first formulation learns distinct quadratic function, the second learns symmetric hinge, and the third learns multi-layer perceptron. These functions have distinct parameterizations for each latent feature and each in-distribution class. The out-of-distribun logit is held fixed at zero: 1x+1 (xx) O.
(quad)) k.m
(Zm's m. Uk 1/.m)) = O.k.m
(zm - [k,m)
(11)
2e 2v m
Jk.m (Zmi Okk,m Uk.m, pk.m) = O.k.m 1m Ug,
(12)
where A is the logit class index and m the feature dimension so that the total unnormalized logit, (z), is (z) Em ft.m (2m). The separable MLP is constructed by concatenating a learned vector per feature and class to each 1-dimensional feature and constructing a standard MLP that operates on that augmented state. This formulation provides the greatest flexibility as it leverages all the benefits of the universal approximator theorem (Hornik) 199T). Unfortunately, we find the MLP version difficult to train in practice, often resulting in degenerate solutions. Fixing the OOD logit at zero provides a fixed reference point for the other logits. We interpret this as each latent feature voting on how in-distribution the example is per-class basis.
5.3 INTEGRALS
The cross-entropy, CE(j(xx,, y), between the normalized model output, (xx). and the true labels, y, is commonly used to train classification models. In this section, explore global and local integrals of the cross-entropy loss used to train most classifiers (see Eq. 3) when ý is given as the composition of the softmax function, a separable function and bijective function, e.g., (f (f(())))) We can express this as ming Ex E CE()(x),,) y) owing to the finite number of elements in T. Ideally, we would minimize the model cross-entropy over all possible examples in D. We accomplish this by defining the subspace, ((e) c D, that has a given label and corresponding conditional distribution, p(x(c)
Exxptxle) CE (ú(x),
pxxtp( [log(ú((x)) - pio)
log(ūa(x)) p(x|c)dx.
(13)
is not possible to represent (x) as separable network due to the softmax operation but it is possible to parameterize the unnormalized logits separable network. Substituting this parameterization
into Eq. 13 and transforming to the latent space yields bound for the expected cross-entropy
/e Se,m (amm)emD (2mmmmml lo)dd)m
Ex-p(iie) JCE (f(x).)) &
Σ
(14)
Σ K+1 M
+ loog
II
exp exp(fa((z)) Pa euleddzn
j=1 R=1

---

Published as a conference paper at ICLR 2022
See Appendix E for full derivation. We will utilize this formulation in conjunction with contrastive prior to supervise OOD examples, Sec. 5.3.1 and with a local prior to enforce consistency, Sec. [33.2]
5.3.1 OUT-OT-DISTRIBUTTON SUPERVISION
We can utilize the cross-entropy integral in different ways by choosing the label we would like to apply over a domain. For example, we can integrate the cross-entropy over the OOD latent space, Ut with the true label fixed to the reject class (cK++),, and the latent distribution over codes is the contrastive distribution (Sec. 5.1
(()): using Eq. 14 with fK+1.n -k+ =0:
K
M
CGLBI. = log
II
/ Ju
exp (fj.m(zn)) Bn(2n)dz,
(15)
In effect, Eq. 15| discourages OOD data from being labeled as any in-distribution label. Since the data and contrastive distributions overlap, the model could degenerate and always make OOD decisions; however, this would be in conflict with the standard cross-entropy loss applied to each example in the training So long as LGLBI, is not weighted too strongly, the model achieves equilibrium by making OOD predictions over regions where the contrastive distribution is more likely. In effect, we supervise OOD training without requiring any OOD data.
5.3.2 LOCAL CONSISTENCY
We may also perform integration over neighborhoods of data points Tj and utilize that point's label over the entire region. This integral serves to improve consistency around observations. Adversarial attacks (Goodfellow et al. 2015 Madry et al. 2018)) are often constructed as perturbations on real data points and adversaiial training finds one such point and includes it in the training set. We can interpret local consistency integrations as form average adversarial training. We reformulate the integral to be centered around particular datum, Xa and its class label, y
LL = - Σ uh log(a (fa(H(x)) + v))) (vvvddv
(16)
complication resulting from local integration is how to select the local distribution, pv (v), and the neighborhood, V. There are many reasonable choices depending on the goal of the local integration. For simplicity, we choose each V. [-8, e] and pm (Fm) to uniformly distributed.
5.4 Loss COMPONENTS
The final training loss used to evaluate the model composed of different combinations of the standard cross-entropy loss over class predictions, LCE En x 1/. log (ū), the negative log- ligeliiel loss log(pz
log xx and the integration losses, LGLBL and L.CL. We always include LCE and LNLL- Based on results from previous hy hybrid networks (Chen al., 2019 Nalisnick et al. 2019b), we weight the negative log-likelihood by 1/M, which is analogous to using bits per dimension and introduce an additional weight over the cross-entropy, A, which controls the relative importance of the generative and classification tasks. When included, each integration penalty shares the same weight as Lce with additional binary weight,
in-distribution data points, LNLL
1 Liotll M fNLL + ALEEE + GOLBLCOLBL + (LCL
(17)
RELATED WORK
Noise Contrastive Distributions Noise contrastive methods introduce a distribution that is distinct from the data distribution. The constrastive distribution can be learned and provides a mechanism to supervise the model to discriminate between true and contrastive samples (Gutmann & Hyvärinen 2010). This forces the model learn discriminative statistical properties of the data. We utilize this idea to inform the null-space over which we will integrate. We fix the data and contrastive distributions and learn the bijective map between input and latent spaces.
7

---

Published as a conference paper at ICLR 2022
Hybrid Models Normalizing flows family of flexible, tractable likelihhod estimators based on the application of learnable, bijective functions] Dinh et al. (2017); Grathwohl et al. (2019). These methods have shown strong performance, both as likelihood estimators and generative processes. Recent work has coupled advances in bijective networks to construct invertible feature extractors that are capped by more conventional classifier. The combination bijective/classiiee structure called hybrid models and show reasonable performance as likelihood and discriminative models Chen al. (2019); Nalisnick al. (2019b). Unfortunately, the discriminative performance of these models is lower than traditional methods. We utilize hybrid models with constraints that enable tractable integration. Other hybrid models architectures are less restricted but prohibits practical integration.
Out of Distribution Detection OOD detection is a challenge for many modern ML algorithms. Recent work has demonstrated that OOD data can achieve higher likelihoods than the training data Hendrycks et al. (2019); ; Nalismick et al. (2019a). Several recent methods have been developed to detect OOD examples inc including Maximum Softmax Probability (MSP) [Hendrycks & Gimpel| (2017).
et al.) (202I). ODIN Liang et al. (2018), and Certified Certain Uncertainty (CCU) Meinke & Hein (2020).
Outlier Exposure (OE) Hendrycks et al. (2019), Multiscale Score Matching J
Lee et al. (2018) utilize a GAN trained with a classifier to produce examples near but outside the training distribution. These methods are constructed specifically for OOD detection whereas our method is applicable to a variety of problems.
Semi-supervised Learning Semi-superve superviee learning is a burgeoning research area that learns from a large data corpus when only small subset are labeled. The problem setting is very pertinent as it is often easy to acquire data examples but can be extremely time consuming to create the corresponding labels. Many modern methods
achieve very strong performance with very few labels Zhang et al.] (2018); Chen et al. (2020). However, most of these methods rely on domain- specific augmentation strategies that are difficult to replicate in new data regimes, i.e., for non-image data. Fortunately, methods such as SSVAE Kingma et al. (2014) and VIME Yoon et al. (2020) are domain agnostic. These methods are built exclusively for semi- -upe--ueeeeseed learning but we only require an additional regularizing penalty to achieve comparable performance on tabular datasets.
7 EXPERIMENTS
All models were constructed using PyTorch Paszke et al. 2019) trained using PyTorch- Lightning (Falcon 2019), utilized bijectors and distributions from Pyro (Bingham et al.] 2018). and were trained using Adam (Kingma & Ba, 2015). We assess the integrable model's perfor- mance in a semi-supervised regime and against O0D examples. See Appendix B and C for ad- ditional experiments and training details.
ThIM AR DAM
(a) Class colored, (b) Z neighbor- (c) Z neighbor-
green OOD
hood
hood
of
Cl
G
7.1 SPIRALS
We construct a synthetic, 2D dataset composed of three intertwined spirals, see Fig. 4a| Suppose that, due to an unknown sampling bias, we only observe two of the three arms (missing the green arm) and constructed our model as two-class problem with a third, reject class.
(d) ply = 0)
(e) ply = 1)
(I) p(x E OOD)
Figure 4: Three-arm spirals results
We sample points in dense 2 2-dimensional grid at twice the range of the data in both dimensions and evaluate the probability that each point belongs to the two known classes and the OOD class. Figures (4d]and 4e/lluusle the probability of belonging to the two known classes and Fig. 41 contains the probability that each point is OOD. Red and white values indicate high and low probabilities, respectively. The model successfully identifies the two in-distribution regions. Unsurprisingly the model also identifies data outside of the training data range as being OOD and, impressively, identifies the unobserved spiral and the region between spirals as being OOD.
8

---

Published as a conference paper at ICLR 2022
Finally, sample square box (orange) around a data point (blue) in the latent space and invert the box to assess the appropriateness of the neighborhood the input space and plot the result, Figs. 4b and 4c As expected, the bijector converts the latent Euclidean neighborhood into semantically meaningful neighborhood. Had we included the local cross-entropy integral in this training process, all points within the orange boundary would have received the same label as the data point.
7.2 OUT OF DISTRIBUTION DETECTION
We test how our models respond to OOD examples when trained with global integrals over the noise- contrastive prior and consistency integrals and compare to methods designed cfor OOD detection. See Appendix B.3 for standard performance and Sec. 6 for discussion of the baselines. Table contains the AUPR for the various methods against similar but different datasets (see Appendix B.4 for the AUROC). We juxtapose SVHN vs CIFARIO and MNIST vs FMNIST plus Extended MNIST (EMNIST) (Cohen et al. 2017). We include a separable, hybrid model without integral regularizations (hybrid) and the same model with regularizations (Int. Reg.) illustrate the utility of learning over hypervolumes. When MNIST or FMNIST are the in-distribution set, the regularized, integrable network performs on par with the best baseline methods. SVHN and CIFARIO show reasonable OOD performance but are not as strong as the baselines. This is not surprising since the integrals rely on a reasonable estimate of p(x), which both datasets have failed to achieve, Table
Table 1: Area under the PR curve (percentage).
Out
GAN
ODIN
MSMA
OE
CCU
Hybrid
Int. Reg.
FMNIST
99.4
98.8
99.9
99.9
93.7 + 6,8
99.7 = 0.17
MNIST
EMNIST
84.5
78.4
91.4
84.3
97.2 + 1.9
99.8 = 0.03
MNIST
99.9
99.2
80.8
97.0
98.3
63.9 + 8.3
95.4 = 0.04
FMNIST
EMNIST
100.
99.3
98.6
99.1
93.8 3.5
98.8 0.47
SVHN
CIFAR1O
98.6
97.3
92.5
100.
100.
71.8 2.0
76.8 1.4
CIFARIO
SVHN
80.5
92.7
99.0
98.5
97.5
84.6 + 0.8
87.0 + 2.1
7.3
SEMI-SUPERVISED LEARNING
We apply the local integral the sepa- rable hybrid model and train several tabular datasets with 10% of the labels augmentation strate-
Table 2: Tabular dataset semi-supervised accuracy (%).
SSVAE
VIME
Local Int.
and domain gies are unavailable. These models do not utilize the OOD class or global penalty. We apply pseudo-labelling with thresholds of 0.9-0.95. Table 2 compares the perfor- mance of the integrated hybrid models to several standard (non-image) semi-supervised baselines on flat MNIST (MNIST, flattened a vector), MiniBooNE Bazarko (2001), and HepMass Baldi et al. (016)). We utilize CNF Grathwohl et al. (2019) as the bijector and the hinge as the classifier. We see that the integrated model achieves similar-to-bett performance than the other methods on all datasets, showcasing the generality of integrated regularizers in different problem spaces.
Flat MNIST
88.9
95.8
94.9 0.16
MiniBooNE
92.2
91.7
93.5 = 0.074
HepMass
83.1
82.0
85.4 + 1.2
8 CONCLUSIONS
In this work, we develop architectures that enable tractable integration over the data space. We demonstrate that the ability to supervise regions and not isolated points encourages the model to learn better representations and be less likely to degenerate. We consider several formulations that allow us to regularize the model's behavior based on consistency and contrast without relying on augmentations of and sparse comparisons between finite collection of data points. We experiment with the various integrals to obtain promising -utto--distriuution detection. Through now tractable integrals, this work enables future methods and applications for learning over continuous regions.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTTS
This research was partly funded by grants NSF IIS2133595 and NSF 2113345 and by NIH IROIAA0268790IAI.
REFERENCES
Faruk Ahmed and Aaron Courville. Detecting semantic anomalies. In Proceedings of 34th AAAI Conference on Artificial Intelligence, 2020.
Samet Akcay, Amir Atapour- Abarghoib,,ghouei,, and Toby P Breckon. Ganomaly: Semi- Semi-supervised anomaly detection via adversarial training. In Asian Conference on Computer Vision, pp. 622-637. Springer, 2018.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 274-283. PMLR, 10-15 Jul 2018. URL URLhttp: 7/proceed ngs. ml w.w..pp.s.............. /v/as//v8//// aha ye 8a html
Pierre Baldi, Kyle Cranmer, Taylor Faucett, Peter Sadowski, and Daniel Whiteson. Parameterized neural networks for high-energy physics. The European Physical Journal C, 76(5), Apr 2016. ISSN 1434-6052. doi: 10. 1001140000pcc10002-01649999 URL http:// doi org/10
1140/epjc/s 0052 016--099--44
Andrew O. Bazarko. Miniboone: Status of the booster neutrino experiment. Nuclear Physics B Pro- ceedings Supplements, 91(1-3):210-215 Jan 2001. ISSN 0920-5632. doi: 10000///02005632200 00943-9. BtRhttt:://dk www.ooooooooooooooooo 10.
5632 (00) 00943-9
Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman, Pyro: Deep Universal Probabilistic Programming. Journal of Machine Learning Research, 2018.
Christopher M. Bishop. Pullern Recognition and Machine Learning (Information Science and Statistics). Springer-Verte Verlag, Berlin, Heidelberg, 2006. ISBN 0387310738.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Bengio, Wallach, H. Larochelle, K. Grauman, N. Cesa- Desa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates. Inc., 2018. 69386f 6bb (423888a...ooe6332222288833333-3
httpttppps.//pccceiiintt nww.ips cttpurrpssccppppppeee. httpp/aape///////11L....
pdd
Ricky Т. Q. Chen, Jens Behrmann, David K Duvenaud, and Joern-Henrik Jacobsen. Residual flows for invertible generative modeling. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d'Alché-Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Cur- ran Associates, Inc., 2019. URL https: http::/pppp//pprooeeeeiiioe neur iprriips.. ps wttpccpeppppp..//./// 955548
fil 1125005555222200055550000rrrr330
.pdf
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and André van Schaik. EMNIST: an extension of MNIST to handwritten letters. arXiv preprint arXiv:1702.05377 2017.
Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE
Signal Processing Magazine, 1416)141 2012.
10

---

Published as a conference paper at ICLR 2022
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. [tRL:tttt:///peeeeeeee net/ httpummmmdiipppeee d=
J.R. Dormand and P.J. Prince. A family of embedded runge-kutta formulae. Journal of Compu- tational and Applied Mathematics, 6(1):19 26, 1980. ISSN 0377-0427. doi: https://doi.org 10.10160071105000(88900000111110 atciccll//1100777555333333 http://ppp//wwsseeeeeeeee....oo cwt.scienccc/////occcccccco
et. al. Falcon, WA. Pytorch lightning. GitHub. Note: http:///giiibbbooom//yffoooiihip lightming, 3, 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. httpppee//1///1eeeeeeeee http:////.....e.e2222..00023a1a.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL ht
arxiv. ordiv.oor//aao abwp//bbs1442... 1572
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable reversible generative models with free-form continuous dynamics. In International Conference on Learning Representations, 2019. URL /tttt:::/
net forum ric=rJxxkKCCc]
Michael Gutmann and Aapo Hyvärinen. Noise-contrasiin estimation: new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington (eds.). Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Reseurch, pp. 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL / httt:::/prooeeiingggn...... wwwpeediiingmm......mmm press/pre//vv// wttp://///////////////a./me////. gutmann10a html
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. Machine learning with a reject option: A survey, 2021.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and distrriibtt ex- amples in neural networks. Proceedings of International Conference on Learning Representations, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019. URL https:// openreview wtwp://wwwwetttru.ummmmeeeea hid.ouunnnddkxxxxcccc.x.x...
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 1991. ISSN 0893-6080. doi: http:://oooonm///...000022211000
URL https
//www.co www.:ssceeeeiiiieoo rect. wtt.rreeee...cereeeeeeeeeeeeeeee
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, ICLR 2014, 12 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URLhttp
//axxiv. org. abs 11112 6980
Diederik P. Kingma. Danilo Jimenez Rezende, Shakir Mohamed, and Max Welling. learning with deep generative models. CoRR, abs/1406.5295. 2014. URL http: //arxiv
org/abs/1eg 4 5298
11

---

Published as a conference paper at ICLR 2022
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions, In Advances in Neural Information Processing Systems. pp. 10236-10245, 2018.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, CIFAR-I0 (Canadian Institute for Advanced Research), 2009, URL
http:/ www. cs. toronto
cirr. html
Dong- Lee. Pseudo-label: The simple and efficient semi-supervises learning method for deep neural networks. ICML 2013 Workshop Challenges in Representation Learning (WREPL), 07 2013.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribtion samples. In International Conference on Learning Representations, 2018.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2015 8. httthhttss:///eennee
net. mnet/forumt forum? 1d= VGkIXRZ
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https:/:
/wtp//ppeervveeeeeeee httpps://oppererritittttt net. forum?
TJZIBFZAb
Ahsan Mahmood, Junier Oliva, and Martin Andreas Styner. Multiscale score matching for out-of- distribution detection. In International Conference on Learning Representations, 2021. URL
https: http:///eppeeeeee.e/e/ee/. net/ wwt.neeennt/eeeeee wwwpmmiidxoooiaeeeeeeeod
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. 2020.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of- distribution inputs to deep generative models using typicality. arXiv preprint 1oXiv:1966.0200 2019a.
Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayanan. Hybrid models with deep and invertible features. Proceedings of the 36th International Conference on Machine Learning. 2019b. URLptttp://axxi owtesiiiirrr///ooo abs 1902. 02767
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Junier Oliva, Avinava Dubey, Manzil Zaheer, Barnabas Poczos, Ruslan Salakhutdinov, Eric Xing, and Jeff Schneider. Transformation autoregressive networks. In International Conference on Machine Learning, pp. 3898-3907, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL style
www.eeessss.rrrisscc.co.m httppppr.ccpppprr///ppp leaning library...
9015-p0ttoo aytorrchan--ipeerr
pdf
G Peter Lepage. A new algorithm for adaptive multidimensional integration. Journal of Computational Physics, 27(2):192-203. 1978. ISSN 0021-9997 doi: https://do.orr//I
com/ science/ ctt.c//ciieertccooccccceeeeeee
12

---

Published as a conference paper at ICLR 2022
L.F. Shampine. Solving odes and ddes with residual control. Applied Numerical Mathematics, 52(1): 113-127, 2005. ISSN 0160-01681111 doi: http://0o.ory//.00000o.ammmmmm.0 URLhttps
/tt. www.ssciencedirec..ccm com/ http:///cca......a.//////////000
David Silver, Julian Schrittieeser Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3354 359, 2017.
W.A. Strauss. Partial Differential Equations: An Introduction. Wiley, 2007. ISBN 9780470054567.
URL htt.://g.//oooss..ooooe.o//ooo/r
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Tumee, and Aleksander Madry. Ro- bustness may be at odds with accuracy. In International Conference on Learning Representations, 2019. /ttppp/ppps///peeeeeeeeee.e/ee/p
Ch. Tsitouras. Runge-kutta pairs of order 5(4) satisfying only the first column simplifying assumption. Computers & Mathematics with Applications. 62(2):770-775, 2011. ISSN 0898-1221. doi:
002.002. URL hwwpttttss/..ssceeeeeeeeeee wwtp://wwwwwe.weeeee........o..e com/
Eric Veach. Robust Monte Carlo methods for light transport simulation. Stanford University, 1998.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Rashion-MNISTT novel image dataset for bench- marking machine learning algorithms, 2017.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh- Sunh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. 2020. URL /ttps:://penreeeeew.eeee net/forumm
Id-ByeLIR4FVS
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self- and semi-supervised learning to tabular domain. In Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, neurips
pp. 11033-11043, Curran Associates, Inc., 2020. URL f53800655330000 oat.aaacc...aaaao00000000000-aae
Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chan- drasekhar. Efficient GAN- anomaly detection. In 6th International Conference on Learning Representations, ICLR 2018, Workshop Track, 2018. htRp: wwt.abs..raa////////ooooo.o
06222
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https
/ooenneview net http://www..etorutmteeeceecee
13

---

Published as a conference paper at ICLR 2022
A PROOFS
A.1 ADDITIVELY SEPARABLE FUNCTIONS
Theorem A.1 (Additive Independent Integration). Given an additively separable function, f(v), an independent likelihood, p(v) I Pun (Um), and domain, D2 Dvy "D DuMs
M Σ m=1
( Dum
Ev-p(v) [f(v)
fm (Um) Pm (Um) dum
(A1)
Proof.
J fn (Un) Pmu (Um) du
fn (Un)
if n # m
(A2)
V Sn (Un) Pn (un) dun
else
M
M
Evwppt) [[(v]] /
Σ
Jn (vw)
II m=1
Pm (m)(
dv
(A3)
x=1
M
= Σ / Jn (vn)
II m=1
Pm (Um) dv
(A4)
Substituting Eq. Al into Eq. A3 reduces the M M-dimensional integral over independent likelihoods into a 1-dimensional integral and completes the proof.
A.2 MULTIPLICATIVELY SEPARABLE FUNCTIONS
Theorem A.2 (Multiplicativee Independent Integration). Given multiplicittively separable function, g(v), an independent likelihood, p(v) I Pm (Um), and domain, D. Dv, ... Dussi
M
Evpp(y) [g (v))
II m=
9m (Um) Pin (Wm) dua
(A5)
Dum
Proof. Since each component 9n matched to a component in Pm. the two products can be recombined to isolate like-dimensions, Each dimension can then be integrated independently.
M dv II sm((on))
Af II Am=1
E Ev)n(v) [v)] -
Pm (Rm))
ff /
(Mi on (v (on()))))))) dus ... dudu
II
M
/ (( ... gna(es))pa(aaa)dde
B ADDITIONAL EXPERIMENTS
All image datasets were trained using Glow as the bijector with the number of levels adjusted for the size of the image. MNIST and Fashion MNIST were trained with Adam for 50 epochs with an exponentially decaying learning rate of = 0.75. SVHN and CIFAR10 utilized a similar training procedure but with 75 epochs and slower exponential decay of y 0.99. We find that utilizing an average instead of sum over separable features and ooftthhresholding at - improves training stability.
14

---

Published as a conference paper at ICLR 2022
B.1 ADVERSARIAL ROBUSTNESS
We test the efficacy of the local cross-entropy integral by utilizing a synthetic dataset from (Tsipras al. 2019). The data is constructed via
+ y
w.p. P w.p. 1- 32 ZM+1 N(n,, 1)
y~ (-1, +1},
21
(B1)
and the adversarial examples are constructed analytically by, essentially, flipping the sign of 17. We utilize this synthetic dataset to test adversarial robustness because it provides upper bound on the adversarial accuracy relative to the standard accuracy which allows us to quantify our performance with respect absolute. Additionally, because adversarial examples without relying on an optimization procedures, there can be no concerns that this defense relies on obfuscated gradients (Athalye et al. 2018). We refer the interested reader to the original paper for thorough discussion.
be constructed exactly
We choose M = 10, = 0.95, and = 2/VM. For these choices, strong classifier will obtain near perfect standard accu- racy with zero adversarial accuracy while a robust classifier will obtain 95% standard and adversarial accuracy. We use similar architecture to that found in Sec. C except we utilize ten blocks instead of five. We train one model with only the standard cross- entropy and negative log-likelihood losses and a second model with the additional global and local cross-entropy integration loss. We find that the model with only the standard losses achieves reasonably good standard performance and fairly poor adversarial accuracy. The model with the additional integration losses contains considerably better adversarial performance, nearing the upper and, necessarily lower, standard accuracy bound. Table summarizes our results and the ideal performances.
Table 3: Std. & Adv. Accuracy
Std.
Adv.
Accurate Robust
100.0
0.0
95.0
95.0
Baseline Integrated
97.9
27.5
94.5
90.8
B.2 TOY SEMI-SUPERVISED REGRESSION
To illustrate the utility of global regularizations, we construct one-dimensional, semi-supervised problem with N (0, 4) and y tanh(x). We keep all values of x but remove y values corresponding to negative x during training. Unlike the standard semi-
Table 4: Semi supervsee integration regularization
[1] =0
Standard
Integrated
Sup. MSE Unsup. MSE
1.195e-5 +'2 1.882e-5
2.487e-5 + 1.769e-5
1.399 + 1.198
0.06187 + 0.04538
E [g]
0.1041 = 0.08582
8.570e = 4.150e-4
supervised problem, the missing labels are not random but are the result of limitations or bias in the data collection process. We train two models that are identical except that the integrated model includes penalty over E [S2 (ÿ (x))] based on foreknowledge that the average value is zero. Table shows how the models perform over the supervised and unsupervised regions. The inclusion of the
integration regularizer allows the model to reason about regions that it has not directly observed and has decreased the error in that region by two orders of magnitude.
15

---

Published as a conference paper at ICLR 2022
B.3 STANDARD PERFORMANCE
We test the impact of integrable models on OOD per- formance against several standard image classification datasets: MNIST (Deng] 2012), Fashion MNIST (FM-
Table 5: Standard accuracy and bits-per- dimension for different datasets
NIST) (iiao et al. 2017) SVHN (Netzer et al. 2011). and CIFARIO (Krizhessky et al. 2009). See Appendix B for architures, distributions, and training parameters. Table contains the validation standard accuracy and bits per dimension for all datasets with all integration regularizers. The upper portion of the table is averaged over 10 runs and contains reject option in the separa- ble model. The lower portion a single run without reject option any integrated regularizers. We find that the model performs reasonably well for MNIST and large degredation in performance for SVHN and CIFAR10.
Acc.
BPD
MNIST FMNIST
98.3 + 0.2
2.54 + 0.13
88.1 + 2.3
4.76 + 0.64
CIFARIO SVHN
73.4 1.8
5.62 + 0.76
89.9 + 0.7
4.14 = 0.08
CIFARIO*
76.5
3.78
SVHN*
90.0
2.24
FMNIST but the integrated losses The removal of these losses produces similar accuracies but much-improved BPD, consistent with the hybrid network results reported by ResidualFlows (Chen et al. 2019) when Glow is used.
B.4 OUT OF DISTRIBUTION DETECTION AUROC COMPARISONS
Table 6: Area under the ROC curve (percentage)
GAN
ODIN
MSMA
OE
CCU
Hybrid
Int. Reg.
FMNIST
99.4
98.7
99.9
99.9
93.6 + 6.5
99.8 + 0.12
MNIST
EMNIST
92.8
88.9
95.8
92.0
77.7 = 12.6
98.2 + 0.28
MNIST
99.9
99.0
82.6
96.3
97.8
69.8 9.2
95.4 + 3.1
FMNIST
EMNIST
99.9
99.3
99.3
99.5
63.4 + 18.2
87.8 + 4.4
SVHN
CIFARIO
96.8
95.9
97.6
100.
100.
87.0
91.3 + 0.6
CIFARIO
SVHN
83.9
96.7
99.1
98.8
98.2
73.5 + 1.8
78.85 + 2.7
B.5 INTERPRETABILITY
The separably model and independent latent dimensions allows us to reason about how the model is making decisions in the latent space. Unfortunately, for most bijectors, it is not possible to carry this interpretability back to the input space. Figure 5 demonstrates the final state of the model trained on Fashion MNIST for several features with respect to the latent distribution, per in-distribution class and juxtaposed with the out of distribution data. Specifically, the top row contains the logit components, fr.m. learned by the separable network, color-coded by class; the middle row contains the distribution of each class (colors matched to logits); and the bottom row contains the distribution of the in-distribution data (blue) and OOD data (red). The top row illustrates how the classification network makes its decisions per feature over the distribution presented in the middle row. We see that the logit components map well to the distribution of the data in each feature and provides some intuition for how certain the classifier is over a feature. This demonstrates how this architecture allows for reasonable interpretibility from the latent space. Any value above zero (the dotted black line) is considered in in-distribution and the most likely class is the line with the greatest value at that point. The features were chosen to demonstrate the diversity of the learned solution over features. Generally, the data maps reasonably well to the bimodal distribution, though we do occassionally see mode collapse as in Fig. 5e) Fortunately, in these cases the logits tend be fairly uninformative and only introduce a small bias. Figures 5a| through show heavy overlap with one of the two clusters but not the other. While we do see some diversity amongst the in in-distribution classes that overlap with the OOD data the "bags" (gray) and "sandals" (cyan) class overlap most often. Finally, Fig. (dd demonsttrss a latent feature where the OOD data falls in the region between the two data components.
show a common trend where the OOD data has
16

---

Published as a conference paper at ICLR 2022
(a) Feature
(b) Feature 2
(c) Feature
(d) Feature
(c) Feature 5
Figure 5: Each subplot corresponds to single latent feature. The top and middle rows contain the unnormalized logit components and distributions per class, with matching colors. The bottom row compares the distribution of Fashion MNIST (blue) and MNIST (red). x-axis shared across all rows.
B.6 MNIST LEAVE-ONE-OUT
Following (Ahmed & Courville 2020), we test the robustness of our method against semantic OOD examples by training models on all but class from MNIST (Deng) 2012) and assessing OOD performance on the held out class. We repeat the process for each class and report the AUROC and AUPR. In all cases, train using ten different random seeds and report the average performance and standard deviation across seeds that do not degenerate against the in-distribution validation set. Table 8 and Table 7 contain the results of our method using both integrated losses compared to several relevant basclines. Results from Efficient GAN-Basee Anomaly Detection (EGBAD) (Zenati et al. 2018) and GANomaly (Akcay et al. 2018) are taken by estimating the results from figures in both works. The Semantic Anomalies (Ahmed & Courville 2020) results are obtained by executing the official version of the code using the rotation auxiliary task with two different batch sizes (128 and 256). The Semantic Anomalies results in both tables are the best across both batch sizes and detection methods (ODIN (Liang et al. 2018) and MSP (Hendrycks Cimpel 2017 based on the AUPR.
We see that the Semantic Anomalies generally achieves the best AUROC across all digits. The integration often achieves the best AUPR but struggles with certain held out digits. In particular, the integration performs significantly worse on the "4" and "9" digits. This is reasonable confuser due to the similarities and variability of the two classes. These results indicate that the integration penalties helpful for inducing the model to detect semantic anomalies.
Table 7: MNIST leave one out: AUPR (%)
Method
0
4
EGBAD
78
29
46
SA: Rotation
89.24
83.57
78.20
66.33
90.45
Global & Local
95.26 + 1.98
89.67
4.33
86.02
3.19
93.41
1.88
69.89 + 7.02
5
9
EGBAD SA: Rotation
43
57
35
54
35
83.38
75.57
95.19
68.84
84.88
Global & Local
87.76 + 2.1
91.7 + 2.84
83.43 + 3.72
87.5 2.91
72.9 + 7.23
C TRAINING AND ARCHITECTURES
C. SPIRALS
The bijective layer is composed of five blocks where each block contains an affine-coupling layer (Dinh et al.) 2017) and an invertible fully connected layer. The data distribution is bimodal over both latent dimensions with means at =1 and standard deviations of 0.4. The separable network is composed of quadratic functions. We train the model discussed, using the standard cross-entropy
17

---

Published as a conference paper at ICLR 2022
Table MNIST leave one out: AUROC (%)
Method
0
4
EGBAD
78
29
45
GANomaloy
88
65
95
79
80
SA: Rotation
98.85
98. 17
94.60
94.22
98.23
Global & Local
95.93 = 1.91
89.32 + 4.76
84.83 + 3.94
94.25 + 1.25
74.49 + 6.98
5
6
9
EGBAD
43
57
36
GANomaloy
855
85
68
85
55
SA: Rotation
97.21
92.52
99.16
92.69
97.02
Global & Local
89.47 + 2.32
93.11 + 2.98
83.45 + 4.01
88.42 + 2.97
75.66 + 5.65
loss for each observed point, the negative log-likelihood over the input space, and the global cross- entropy integration with respect to the contrastive prior. The learning rate set 0.001 with standard weight decay of le-4 over 20 epochs using Adam (Kingma & Ba 2015).
C.2 FASHION MNIST
The bijective layers composed of the Glow (Kingma & Dhariwal 2018) architecture with two levels composed of 32 steps and 512 channels. We utilize an open-source Glow implementation 2018) bijective operators. The data distribution is bimodal over all
wrapped in Pyro's (Bingham latent dimensions with means at 1 and standard deviations of 0.5. The noise constrastive distribution standard Gaussian. The separable network composed of hinge functions (see Sec. 4.3). We utilize batch size of 256, learning rate of 1.5e-4 over the bijective layers, and a learning rate of 1.0e-3 over the separable layers using Adam (Kingma Ba 2015). Both learning rates are exponentially decayed at rate of 0.75 epoch. Standard weight decay is applied with weight of le-4. The network is trained for epochs. The adversarial attacks are performed against the model at the epoch with the lowest validation cross-entropy loss.
D APPROXIMATE EXPECTED CROSS-ENTROPY OVER A DOMAIN
We desire the expected cross-entropy, CE(j(x), yc), over a domain De where Ye the desired probability vector over classes, j(x) is the model's prediction
E TRK from x E R.M and is composed of bijective function, h, separable function, fx (x) = - fr.m (xm). and the soft-max function, ar, such that (xx = a (f(h(x))) If we are attempting to regularize the model to produce single class label over D., then y will correspond to one-hot vector. In general, however, y. may be a dense vector with elements yk- The expected cross-entropy is expressed as
ED. (CE()
Σ ys Eo. (log((
Σ
Uk
log(ūx(x)) PD. (xly)dx (DI) Jp.
where PD. (xly) is the probability density function over De conditioned y. We can take advantage of the bijective transform to convert this expression into an expectation over the latent space, 2, and latent domain, Z., which allows to write
Uk ED. [log(Un (x))
Σ un Ez. [og(o([(2))]
Σ yk /
log(an (f(x()(()))))))))))))(()())))( pz, (zly)dz
J2.
(D2)
where 0% is the kth element after the soft-max normalization and pz, (zly) is the independent probability density function in the latent space, e.g., Pz. (zly) IIm Pm (2m). We can then expand
18

---

Published as a conference paper at ICLR 2022
the soft-max operator within the expectation get
]
Σ y/ Rz [og(((((()) Ez
log
Σ
(x)( j=1
Σ k=1
ys Rz. [fx(z]
(D3)
The combination of the separable function and independent densities allows for exact simplification of the second term into
K
M
Σ
Uk Ez, [fa(2)]
y/
Σ
fkim (Zen (onn))pmll )m2d
(D4)
Zm
m=
K
M
1
22
3k
mo (Zm.g)
G
m=1g=1
where we have overloaded mm to correspond to the domain of the mith latent dimension and have approximated the integral via Monte Carlo integration with mm.gfm (2m) and G draws. The first term on the right-hand side of Eq. doejoss not enjoy the same exact simplification. However, is possible to bound the term via Jensen's 's Inequality by moving the logarithm into the sum over outside the expectation operator. We consider the latter case where
K Σ
K Σ
exp((((2)))
exp((((2))
Ez.
log
S log
EZ,
(D5)
Then, we expand the separable function, exchange the order the sum over dimensions and the exponent, take advantage of the multiplicative integral simplification (Eq. A5 approximate via Monte Carlo integration, and utilize the log > exp operator for stability:
M
log Ez, Σ exp((s(2))
= log
Ez.
Σ exp(
Σ
Jam(smi))
(D6)
m=1
K
M
M
= log
dz
II exp(fsm (2m)) II n=1
Pm(zn)
j=l
2
m=1
K
M
= log
Σ
II
exp(Jj,m 1) (Zm) ddzm
j=
Zm
K
M
G
log
Σ
exp exp(fsm(Emma)) (mmm)))))))) log(G)
1=1
m
g=1
K
M
log
exp
log
exp(Jim (Zm..)) M log(G).
j=1
m
Finally, we can substitute Eq. D4 and Eq.
D6
into Eq.
DI
get
K
M
ED. [CE(g(x),
< Σ
yk
Σ
Sk.m (Zm )m (Zm Jdzm
(D7)
1=
mmm
Zm
K
M
+ log
(Zm ddi
1=1
m=1
Zm
K
M
G
Uk
fk.m (Zm.g)
G
k=
K
M
+ log
cxp
log
(xp(f)mn(2mms)) - M log(G)
==1
m=
19

---

Published as a conference paper at ICLR 2022
In the event that ye is one-hot vector (at the cth element) this simplifies to
/
Ep, [CE(Ú(x),
S
Σ
Se.m(Zm
Pm ( (zm cc)d
(D8)
m=1
Zm
log
EI /=11 m=
exp(J3.m (emJ),m(Zmm)pp Pon lc)
where reintroduced the condition over the class within the probability densities Pm (m (c) for emphasis.
20


Reviews:
[{"review_id": "t1IOvkS5fAd", "reviewer": "Reviewer_MsPk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written and relatively easy to follow.  The method combines known methodologies in a relatively straightforward but novel manner to achieve the desired goal, which is well-motivated.  The exposition is clear and the experimental section covers a good number of compelling applications for the method.\n\nOne potential issue with this paper is that the method described simply combines known approaches.  Overall I don't consider this a particularly serious flaw but it is something to consider.", "summary_of_the_paper": "The paper proposes using separable bijective networks - that is, a two-stage network where the first stage is bijective (data-flow either way) and the second stage is separable (multiplicative or additive) - to make it practical to include integrals (on the input) as performance goals.", "summary_of_the_review": "Good paper combining known methods to solve a well-motivated problem with a number of compelling applications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YTogciobafE", "reviewer": "Reviewer_6qXG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method uses a bijective flow network to map dependent input features to a set of independent latent variables. It then applies a separable function on the latent variables to get a loss function (or an approximation). The combination results in an integral that decomposes into separate 1-d integrals across each dimension. This makes it possible to calculate the gradient of the loss function efficiently across the latent domain.\n\nThe empirical results are mixed. The use of regularizers based on the decomposable integrals is shown to be beneficial for OOD detection, but the results cannot always match the baseline (likely due to the constraints on the network structure). The results of experiments with adversarial data (in the appendix) look less interesting and might suggest that the method is not effective against targeted adversarial attacks. The description of the experiments is minimal in the main body of the paper and in parts hard to follow.\n\nNote: I was a reviewer to an earlier version of this work. Compared to the previous version, the writing is improved in the parts and the experiments are expanded and rearranged.\n\nMinor notes:\n- It’s more accurate to present the sample complexity of the separable integral as $O(GM)$ when comparing to $O(G^M)$.\n- Theorem names do not match between the main body and the appendix.\n- The notation in section 5.3 does not match the discussion on separable functions in section 3.2.1. In particular, it is better not to use $x_m$ as the argument (maybe $z_m$ or $v_m$). It’s also better to make the parameters of the network explicit in the formulas (11): e.g. $f_{k,m}(z_m; u_{k,m}\\nu_{k,m}) = \\dots$\n- Add names to the first two columns of Table 2.\n", "summary_of_the_paper": "This paper introduces a hybrid model architecture that makes it possible to integrate a separable loss function across a region of input space. Such integrals can be used as regularizers for robustness near the observed data points (local consistency), and out-of-distribution (OOD) detection in neighborhoods away from the observations. The paper uses these regularizers to train models that are less vulnerable to adversarial attacks and out of distribution mishaps.", "summary_of_the_review": "Originality and significance: The combination of flow networks with separable functions to form tractable integrals is a novel idea (to the best of my knowledge), with potential use in other ML domains. The experiments on OOD detection tasks show parity with SOTA baseline in some but not all the studied cases.\n\nQuality and clarity: The paper is hard to follow in parts, mostly due to the fact that it is not self-contained. The reader is expected to know the background on bijective networks. The experimental section is lacking details in the main body of the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QAGOIdHQcN-", "reviewer": "Reviewer_mYvf", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The writing renders the connection between (\\*) and (\\*\\*) very obscure. After all, commonly-encountered $\\Omega \\circ \\hat y$ in ML/DL cannot be written as the composition of a separable function and bijective function. In other words, it is difficult to discern the relevance of (**) when the goal is to estimate (*).\n\nReading Appendix D helped clarify this confusion. The paper operates under the assumption that the model prediction $\\hat y$ can be written as the composition of a bijective $h$, a separable $f$ and the soft-max function $\\sigma$, i.e., (***) $\\hat y(x) = \\sigma(f(h(x)))$. This is a big assumption and should not be relegated to the Appendix. It needs to be front and center in the main text. Even under this assumption the expected cross-entropy in (D1) still cannot be written as (\\*). The best we can do with this (\\***) assumption is to derive an upper bound on (\\*). This is really quite underwhelming. Give me MC error over a potentially very un-tight bound  any day. \n\nBesides the exposition, I hold reservations about the proposed methodology. It is not clear that it’s a worthwhile to restrict a neural network classifier to be of the form (***) only to arrive at the bound in Equation D8 (which is same as Equation 13 in the main text). ", "summary_of_the_paper": "For input-output pair $(x,y)$, the paper undertakes the task of estimating (*) $E_{x \\sim p(x)} \\Omega(\\hat y(x))$. When $x$ is high dimensional, integration is difficult. Standard ML/DL applies MC to estimate this integral based on an iid sample $(x_i,y_i), i=1,\\ldots,n$. This paper suggests there is a better way. \n\nThe main idea is to recognize that if $f$ is a separable function and $h$ is a bijective function, then (**) $E_{x \\sim p(x)} f(h(x))$ is easy to evaluate. ", "summary_of_the_review": "Although I like the idea that we should pay more attention to integration error, I don't see how we can get around it in ML/DL. The proposed approach of assuming (\\***) is certainly not the way. Furthermore, the function $\\Omega$ in (\\*) that would be encountered in ML/DL cannot be accounted for in the proposed framework unless some type of bound is in D8 is employed. Replacing MC error with an upper bound on a nice loss function as in D8 simply does not seem like a good trade to me.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "-gYozcz66Ri", "reviewer": "Reviewer_PTfm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written and well-motivated. I think that integration over a continuous region -- instead of the more heuristic approach of random perturbation or augmentation -- is a novel and more principled strategy that enhances the robustness of models. For the purpose of tractable integration, using neural networks to parametrize separable functions is a natural approach. \n\nMy main concern, however, is the scope of application of this work. Based on Section 1-6 of the paper, my impression is that the authors aim to keep the method general-purpose. Indeed, when reviewing relevant work on OOD in Section 6, the authors mentioned that \"These methods are constructed specifically for OOD detection whereas our method could be applied to a variety of problems based on the continuous regularizers chosen.\" With this impression in mind, I am slightly disappointed to find that all experiments in Section 7 are related to OOD detection. What is more, the proposed method is (slightly) worse than contemporary OOD detection baslines. For this reason, I find it difficult to assess the significance of the contribution of this work purely based on its performance on OOD detection. To showcase the claimed versatility of the proposed method, I suggest the authors include tasks other than OOD detection, or perhaps explain why only OOD detection is pertinent to the proposed method.", "summary_of_the_paper": "In their submission, the authors proposed a computationally tractable way to perform integration over a high-dimensional region that the data samples reside. This integration technique is applied to OOD detection tasks. \n\n\n\n\n", "summary_of_the_review": "To summarize, I think the current paper put forward an interesting idea (perform computationally tractable integration using separable functions parametrized by neural nets). However, it is unclear (at least based on the current presentation) whether the idea finds practical application in machine learning. The experiment section of this paper focuses on OOD detection, in which case the proposed method performs OK-ish but fails to compete against contemporary baseline methods. Thanks to its generality, the proposed integral method may be useful for other ML applications, which I encourage the authors to demonstrate.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper suggests the use of networks for supervised learning which are composed of a bijective network (e.g. a flow) followed by a separable function. This allows easy integration over the input space, which can be used to formulate novel regularizers (examples given are for local consistency, and for out-of-distribution detection).

The approach is pretty novel, and it's an interesting paper. The reviewers were very divided, however; one reviewer giving it a 1, and another an 8, with the other two reviewers arguing weakly to accept. The "1" took issues with the general formulation, feeling that the necessity to optimize bounds on the true objectives in cases such as softmax regression greatly limits the viability of the work. Personally I disagree with that reviewer's characterizations of the novelty and significance of this work, and I think the OOD detection / classification experimental setting is sufficient to make their point that their approach can be applied in such settings.

In the end I (AC) would agree with the "weak accept" / 6, given the draft of the paper at this time. While I think a few things could be presented more clearly, and I think the empirical evaluation could be more robust (e.g. exploring what goes wrong when the integrated objective functions are included on CIFAR and SVHN), and it would be nice to explore additional applications, I think this is a creative paper which it would be nice to include at ICLR.
Decision: Accept (Poster)