Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes using separable bijective networks - that is, a two-stage network where the first stage is bijective (data-flow either way) and the second stage is separable (multiplicative or additive) - to make it practical to include integrals (on the input) as performance goals.
main_review: The paper is clearly written and relatively easy to follow.  The method combines known methodologies in a relatively straightforward but novel manner to achieve the desired goal, which is well-motivated.  The exposition is clear and the experimental section covers a good number of compelling applications for the method.

One potential issue with this paper is that the method described simply combines known approaches.  Overall I don't consider this a particularly serious flaw but it is something to consider.
summary_of_the_review: Good paper combining known methods to solve a well-motivated problem with a number of compelling applications.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a hybrid model architecture that makes it possible to integrate a separable loss function across a region of input space. Such integrals can be used as regularizers for robustness near the observed data points (local consistency), and out-of-distribution (OOD) detection in neighborhoods away from the observations. The paper uses these regularizers to train models that are less vulnerable to adversarial attacks and out of distribution mishaps.
main_review: The method uses a bijective flow network to map dependent input features to a set of independent latent variables. It then applies a separable function on the latent variables to get a loss function (or an approximation). The combination results in an integral that decomposes into separate 1-d integrals across each dimension. This makes it possible to calculate the gradient of the loss function efficiently across the latent domain.

The empirical results are mixed. The use of regularizers based on the decomposable integrals is shown to be beneficial for OOD detection, but the results cannot always match the baseline (likely due to the constraints on the network structure). The results of experiments with adversarial data (in the appendix) look less interesting and might suggest that the method is not effective against targeted adversarial attacks. The description of the experiments is minimal in the main body of the paper and in parts hard to follow.

Note: I was a reviewer to an earlier version of this work. Compared to the previous version, the writing is improved in the parts and the experiments are expanded and rearranged.

Minor notes:
- It’s more accurate to present the sample complexity of the separable integral as $O(GM)$ when comparing to $O(G^M)$.
- Theorem names do not match between the main body and the appendix.
- The notation in section 5.3 does not match the discussion on separable functions in section 3.2.1. In particular, it is better not to use $x_m$ as the argument (maybe $z_m$ or $v_m$). It’s also better to make the parameters of the network explicit in the formulas (11): e.g. $f_{k,m}(z_m; u_{k,m}\nu_{k,m}) = \dots$
- Add names to the first two columns of Table 2.

summary_of_the_review: Originality and significance: The combination of flow networks with separable functions to form tractable integrals is a novel idea (to the best of my knowledge), with potential use in other ML domains. The experiments on OOD detection tasks show parity with SOTA baseline in some but not all the studied cases.

Quality and clarity: The paper is hard to follow in parts, mostly due to the fact that it is not self-contained. The reader is expected to know the background on bijective networks. The experimental section is lacking details in the main body of the paper.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: For input-output pair $(x,y)$, the paper undertakes the task of estimating (*) $E_{x \sim p(x)} \Omega(\hat y(x))$. When $x$ is high dimensional, integration is difficult. Standard ML/DL applies MC to estimate this integral based on an iid sample $(x_i,y_i), i=1,\ldots,n$. This paper suggests there is a better way. 

The main idea is to recognize that if $f$ is a separable function and $h$ is a bijective function, then (**) $E_{x \sim p(x)} f(h(x))$ is easy to evaluate. 
main_review: The writing renders the connection between (\*) and (\*\*) very obscure. After all, commonly-encountered $\Omega \circ \hat y$ in ML/DL cannot be written as the composition of a separable function and bijective function. In other words, it is difficult to discern the relevance of (**) when the goal is to estimate (*).

Reading Appendix D helped clarify this confusion. The paper operates under the assumption that the model prediction $\hat y$ can be written as the composition of a bijective $h$, a separable $f$ and the soft-max function $\sigma$, i.e., (***) $\hat y(x) = \sigma(f(h(x)))$. This is a big assumption and should not be relegated to the Appendix. It needs to be front and center in the main text. Even under this assumption the expected cross-entropy in (D1) still cannot be written as (\*). The best we can do with this (\***) assumption is to derive an upper bound on (\*). This is really quite underwhelming. Give me MC error over a potentially very un-tight bound  any day. 

Besides the exposition, I hold reservations about the proposed methodology. It is not clear that it’s a worthwhile to restrict a neural network classifier to be of the form (***) only to arrive at the bound in Equation D8 (which is same as Equation 13 in the main text). 
summary_of_the_review: Although I like the idea that we should pay more attention to integration error, I don't see how we can get around it in ML/DL. The proposed approach of assuming (\***) is certainly not the way. Furthermore, the function $\Omega$ in (\*) that would be encountered in ML/DL cannot be accounted for in the proposed framework unless some type of bound is in D8 is employed. Replacing MC error with an upper bound on a nice loss function as in D8 simply does not seem like a good trade to me.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In their submission, the authors proposed a computationally tractable way to perform integration over a high-dimensional region that the data samples reside. This integration technique is applied to OOD detection tasks. 





main_review: The paper is clearly written and well-motivated. I think that integration over a continuous region -- instead of the more heuristic approach of random perturbation or augmentation -- is a novel and more principled strategy that enhances the robustness of models. For the purpose of tractable integration, using neural networks to parametrize separable functions is a natural approach. 

My main concern, however, is the scope of application of this work. Based on Section 1-6 of the paper, my impression is that the authors aim to keep the method general-purpose. Indeed, when reviewing relevant work on OOD in Section 6, the authors mentioned that "These methods are constructed specifically for OOD detection whereas our method could be applied to a variety of problems based on the continuous regularizers chosen." With this impression in mind, I am slightly disappointed to find that all experiments in Section 7 are related to OOD detection. What is more, the proposed method is (slightly) worse than contemporary OOD detection baslines. For this reason, I find it difficult to assess the significance of the contribution of this work purely based on its performance on OOD detection. To showcase the claimed versatility of the proposed method, I suggest the authors include tasks other than OOD detection, or perhaps explain why only OOD detection is pertinent to the proposed method.
summary_of_the_review: To summarize, I think the current paper put forward an interesting idea (perform computationally tractable integration using separable functions parametrized by neural nets). However, it is unclear (at least based on the current presentation) whether the idea finds practical application in machine learning. The experiment section of this paper focuses on OOD detection, in which case the proposed method performs OK-ish but fails to compete against contemporary baseline methods. Thanks to its generality, the proposed integral method may be useful for other ML applications, which I encourage the authors to demonstrate.
review_title: null

Meta-review: This paper suggests the use of networks for supervised learning which are composed of a bijective network (e.g. a flow) followed by a separable function. This allows easy integration over the input space, which can be used to formulate novel regularizers (examples given are for local consistency, and for out-of-distribution detection).

The approach is pretty novel, and it's an interesting paper. The reviewers were very divided, however; one reviewer giving it a 1, and another an 8, with the other two reviewers arguing weakly to accept. The "1" took issues with the general formulation, feeling that the necessity to optimize bounds on the true objectives in cases such as softmax regression greatly limits the viability of the work. Personally I disagree with that reviewer's characterizations of the novelty and significance of this work, and I think the OOD detection / classification experimental setting is sufficient to make their point that their approach can be applied in such settings.

In the end I (AC) would agree with the "weak accept" / 6, given the draft of the paper at this time. While I think a few things could be presented more clearly, and I think the empirical evaluation could be more robust (e.g. exploring what goes wrong when the integrated objective functions are included on CIFAR and SVHN), and it would be nice to explore additional applications, I think this is a creative paper which it would be nice to include at ICLR.
Decision: Accept (Poster)