Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: (Updated score post rebuttal)

The paper proposes a new method AutoInverse in the field of inverting neural networks that incorporates the uncertainty in the modelling of the NFP (natural forward process) though a neural network surrogate. By adding the epistemic uncertainty of the surrogate function (due to mismodelling) and aleatoric uncertainty in the training data (noise in the data), the new method favours points that have low uncertainty, and therefore are a better match to the NFP. Incorporating uncertainty in the inversion loss also results in desirable behaviour, such as 1) enforcing the feasibility in solutions, as points with low feasibility should have high uncertainty, 2) not requiring regularisation techniques, and 3) not requiring careful initialisation. 
strengths_and_weaknesses: 1. Originality

The Autoinverse technique has two components: 1) Replacing the surrogate neural network function with a neural network function that is capable of predictive uncertainty, and 2) modifying the loss function used to find accurate designs given a pretrained surrogate function with frozen parameters.

The first component of this technique is a well-established baseline in literature, and deep ensembles have been very well studied in their ability to accurately model predictive uncertainty in a bunch of different domains and tasks. In fact, the decomposition of the uncertainty learnt using deep ensembles (or other methods that model heteroskedastic regression) into components that model aleatoric and epistemic uncertainty are also well-studied [1]. However, the authors do not cite [1], or discuss any potential pitfalls of modeling noise in such a way (see [2] as an example.)

The second component is the novel contribution for this paper, in my uncerstanding. By using a neural network that is modeling both heteroskedastic aleatoric uncertainty and epistemic uncertainty, this method can use these estimates as auxilliary terms in the inverse cost function, to directly influence the training of the designs. By adding epistemic uncertainty to the inverse cost function, points that have high epistemic uncertainty are downweighted (in a sense), and these are usually points where the surrogate is not performing well or is performing pathologically. By adding aleatoric uncertainty to the cost function, points with high data noise are also downweighted. This simple addition of two terms to the cost function results in quite impressive empirical performance.

2. Quality

In my opinion, the paper is well-written, has good motivations for the problem it is tackling, and has strong ablations that answered quite a few of my initial doubts/hesitations about the empirical strengths of the technique. 

I have some concerns regarding the thoroughness of uncertainty quantification methods studied, and a careful analysis of the strengths and weaknesses of different methods. For example, Deep Ensembles are known to be state-of-the-art in uncertainty quantification, but have much higher compute time and storage requirements than other methods. The authors mention that the longer training time for ensembles is not a concern, as they are trained parallely. I disagree with this point, as I feel like acknowledging the GPU hours required to train a method are quite important, especially with big compute quite easily getting out of hand to obtain SOTA. Even though there are certain groups with the resources to trivially parallelise ensembles, if Autoinverse is to be established as a strong contender, the added compute time, memory footprint, and inference time needs to be acknowledged. 

Another factor here is maybe exploring a few other uncertainty quantification techniques as an ablation in the paper (MC Dropout, more efficient ensembles etc.). I feel like this comparison is important to understand the contribution of adding an aleatoric and epistemic uncertainty term to the cost function, vs the quality of these uncertainty estimates w.r.t the compute required. It might be possible that deep ensembles are overkill for the inversion problem, and a lot of compute is used to obtain uncertainty estimates that might be approximated more cheaply. As the authors mention, it is a very interesting idea to capture the Pareto front between accuracy and uncertainty in an inversion task.

3. Clarity

The paper is quite easy to follow, and the experiments chosen make logical sense. I would argue that it is more important to describe the inversion technique and cost function before talking about uncertainty quantification, as it is maybe more prudent to motivate the exact problem that the authors wish to tackle. In the current ordering, it might seem that the authors use a well-known methodology of uncertainty quantification, and find a usecase that works well. Though this is obviously not true, a reordering of the sections might help mitigate this.

4. Significance

Uncertainty quantification in deep neural network applications is of fundamental importance. In order for DNN methods to be applicable in safety-critical domains, it is very important that we measure their performance, calibration and out-of-distribution robustness. To my knowledge, there has not been prior work that adequately addresses the surrogate-NFP mismatch when formulating the cost function for inversion. This paper's application is quite significant, and I believe that the empirical results show that modelling uncertainty in a principled way can result in multiple benefits. However, I feel like this paper is lacking in terms of adequately populating the Pareto front between uncertainty and accuracy, by proposing a single data point (ensembles). While the current proposed method cannot be argued with in terms of empirical performance, I feel like this paper might be a good place to populate more points here, to establish a better understanding for the community, on how the quality of uncertainty estimates affect the performance of inversion.

[1] Kendall, Alex, and Yarin Gal. "What uncertainties do we need in bayesian deep learning for computer vision?." _Advances in neural information processing systems_ 30 (2017).

[2] Seitzer, Maximilian, et al. "On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks." _arXiv preprint arXiv:2203.09168_ (2022).
questions: 1. I feel like the authors need to incoporate a few more papers in their literature survey regarding the modelling of aleatoric and epistemic uncertainty using a heteroskedastic regression loss. The decomposition shown in the paper has been shown before, in [1]. There has been work also showing that using this formulation can have pathological behaviour [2]. 
2. I found this paper lacking an adequate explanation of the tradeoff between aleatoric and epistemic uncertainty terms. The authors mention that "We observe that with a larger epistemic uncertainty we obtain better results", but I think it's important to have a better ablation here. An easy way of "improving" epistemic uncertainty estimates is to use more deep ensembles, and see how increasing the number of ensembles affects the choice of $\alpha$ and $\beta$ parameters. Similarly, by adding more noisy data, in the "Locally noisy data set" ablation, one can measure the importance of the aleatoric term. I think having these two sweeps in the ablation, along with $\alpha$ and $\beta$, will really help understand the differences between the two terms.
3. Given that the main contribution of this paper is a way of incorporating uncertainty estimates into the inversion cost function, I think in order for the paper to be a good candidate for acceptance, there needs to be a better ablation for different uncertainty estimation methods. I've mentioned more of this in the Significance section.

Minor nits:
- Line 75: "neural"
- Line 112: "push"
- Line 313: "similar"
limitations: I believe the authors should address the additional compute requirements needed for their method. Deep ensembles are the least efficient way of obtaining uncertainty estimates, and can result in significantly more GPU hours for training and inference. These should be addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents an approach to solving inverse problems, i.e. find parameters $\mathbf x$ that explain/produce given observations $y$ when passed through the forward model $f(\mathbf x)$. The approach explicitly models and accounts for the uncertainty of the forward process by learning a surrogate of it using a deep ensemble. The deep ensemble models the forward process as $y \sim N(\mathbb F_\mu(\mathbf x), \mathbb F_\sigma(\mathbf x)$, with $\mathbb F_\sigma(\mathbf x)$ explicitly modelling the uncertainty of the forward process. The   $\mathbb F_\sigma(\mathbf x)$ is then plugged within two inversion methods, the neural adjoint and a tandem architecture, producing their uncertainty-aware variants. As a result when looking for the inversion solution the method will avoid $\mathbf x$ which come with a high uncertainty with the $y$ that they produce. The two methods are evaluated in three benchmarks and demonstrate good inversion performance. 
strengths_and_weaknesses: The paper is rather well written. The main idea of the paper is that when looking for inverse solutions, $\mathbf x$, that could produce a given observation $y$ one should avoid $\mathbf x$ that come with high uncertainty for their predicted $y$, as quantified by the deep ensemble and the estimated $\mathbb F_\sigma(\mathbf x)$ parameter. In fact $\mathbb F_\sigma(\mathbf x)$ can be decomposed in two terms one measuring aleatoric uncertainty and the other the epistemic uncertainty. The latter is the extend to which the components of the ensemble differ with respect to the average prediction, i.e. the $\mathbb F_\mu(\mathbf x)$. A high epistemic uncertainty can be evidence of the $\mathbf x$ strays away from the training distribution and thus the base models produce very different predictions. Incorporating the term in the loss functions keeps the search for an inverse solution away from such high uncertainty regions in which the learned surrogate is probably not a very accurate proxy of the true forward process. 

If I understood correctly the paper only provides a point estimate, i.e. a single $x$ given a desired $y$, and along the above discussion the one with the least uncertainty. To my understanding in solving inverse problems one is also strongly interested in accessing the full posterior of $\mathbb x$, i.e. $p(\mathbb x|y)$, something that is delivered by generative methods such as the invertible neural networks which the authors cite and compare as one of the baselines. There is no discussion, unless I missed it, on the quite point estimate vs posterior distribution approach; this is a weakness of the paper, and one could also say a weakness of the method.  


questions: In relation to the single-point vs the complete posterior distribution it would be useful to discuss how the single point solution fits within the complete posterior distribution, can we make any other statement on where the point estimate x relates with the complete p(x|y) other than that it is an $x$ for which $p(y|x)$ has the least uncertainty?

In the evaluation measures I am not sure I get correctly what the surrogate error is and what the NFP error is. Is the surrogate error $|| f(f^{-1}(y)) - y||$ or is it $||f(\mathbf x) - y||$, in other words does it measure the gap between the prediction of the forward surrogate in the inverse solution, or the gap on paired training instances (x,y). In what concerns the NFP error, is this the $|| NFP(f^{-1}(y)) - y ||$, i.e. the error we get when we pass through the real forward process at the inverse solution? 

With respect to the INN wouldn't a more appropriate way to get the inverse solution be the MAP estimated of the p(x|y) instead of getting empirical samples from it?
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper discusses an approach for inverting surrogate functions, based on neural networks, in a face of (epistemic and aleatoric) uncertainty.
strengths_and_weaknesses: 
Strengths :

- The problem statement: quantification of uncertainty for invertible processes is an important problem
- The idea of modelling uncertainty as outlined in Section 3.1 is a reasonable one
- The authors plan to release the spectral printer dataset; the new dataset would be beneficial for the research community


Weaknesses:
- The writing looks not entirely clear (see the question section)
- Novelty could also be emphasised as it is unclear where the proposed method stand amongst the state-of-the-art models such as variational autoencoders and normalising flows (see q1 in the questions section)

I put the score as 'reject' for now, but looking forward to the rebuttal and hope the authors could address the questions appropriately.

***

The score is updated as the concerns have been thoroughly addressed during the rebuttal (see the discussion below).
questions: 1) Section 2 (Related work) would be expected to place  this work within the current state-of-the-art, however, it does not look like it fully does: how do the state-of-the-art works address the problem of estimation of uncertainty, what prior methods address this problem and what is the closest work which the authors build or improve upon? It is said that ‘In this method, a specialized architecture built upon normalizing flows [9] is trained   simultaneously in both forward and inverse directions leading to a bijective mapping between design and performance spaces. “ but it is not followed up in the following sections.  Furthermore, the similarity between the proposed ensemble architecture and normalising flows is unclear to the reviewer. (Also, on this sentence, the reviewer has probably misunderstood how exactly could the proposed modification of normalising flows be trained in both directions, if the original normalising flows [Resende &Mohamed, 2015] are invertible-by-design?)
2) The reviewer struggles to follow section 3 (Method): what is the precise model architecture, for example? (while there is the code so that the model is reproducible, it is still important to discuss and justify main aspects of implementation) While Appendix gives implementation details (Section 1 of the Appendix), it is unclear what the relation to neural adjoint method is and how does the proposed model build upon it; furthermore, it would be good if the paper references the descriptions in the appendix. 
3) In Eq. 4-5, it is important to clarify, why the optimisation happens with respect to the inputs X, and where and how are the parameters of the model optimised.
4) “Following [27], we take a 3-step procedure for implementing deep ensemble predictive uncertainty. “ What would the three steps be?
5) On the experiments side, it is important to describe the experimental tasks in more detail to make it reproducible (which are not Ava; the reviewer thinks that the paper should describe what exact hyperparameters and settings correspond to the experimental evaluation.
6) It is important to identify (and compare with) the state-of-the-art methods; if it is not possible for whatever reason, there should be a convincing argument why such comparison would not be valid. I accept that Table 1-2 present evaluation against Neural Adjoint method, but just two comparisons on two tasks do not indicate the advantages (and also importantly, restrictions) of the proposed method.
7) (Not as critical as the previous points but still) The concluding remarks state: ''We look forward to see Autoinverse adopted for more inverse architectures, beyond tandem and more optimizations (beyond first-order NA). The Autoinverse cost function is multi-objective. It is highly interesting to capture the trade-off between accuracy and uncertainty using the Pareto front. '' It does not give much summary of the main conclusions of the paper, and also the future work suggestions are difficult to follow (more inverse architectures/optimisations: in which sense? Pareto front: would be interesting to see it, could be actually quite nice to see them as an additional experiment, e.g. in the appendix)

limitations: There is a need to discuss the limitations of the work more; one way to address it would be to use standard benchmarks such as the ones proposed in the standard invertible neural networks literature (e.g. MNIST and CIFAR experiments from Rezende and Mohamed, 2015, or experimental section of Grathwohl et al, 2019); otherwise, it would be useful to state why such experiments are not like-for-like or not possible.

* Rezende D, Mohamed S. Variational Inference with Normalizing Flows. In: International Conference on Machine Learning, 2015 (pp. 1530-1538). PMLR.
* Will Grathwohl*, Ricky T. Q. Chen*, Jesse Bettencourt, Ilya Sutskever, David Duvenaud. "FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models." International Conference on Learning Representations (2019).
review_title: null

Meta-review: metareview: Thanks to the authors for submitting this super interesting work. The reviewer discussion reflected an overall satisfaction with the submission, author responses, and updated manuscript.  The additional experiments directly addressed reviewer concerns, and contributed to increased scores and clarifications in the submission.  Given the clear consensus and reviewer enthusiasm, I recommend acceptance.  Well done! 
confidence: Certain
Decision: Accept