Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper concerns statistical inference in linear systems. A typical approach uses an integral formulation of the problem via Green's function of the differential operator. The approach taken in this paper is to instead use adjoints turning the problem into a linear model and then use a reduced-rank Gaussian process for posterior approximation. In more detail, once the adjoint of the linear operator is found, n (number of observations) equations composing a linear system are set up and the unknown parameters are solved for using the standard least squares approach.

strengths_and_weaknesses: Overall, I think this is a very nice paper, and seems like a clean approach to inference in the class of problems the paper addresses. The solutions produced enjoy stability and are conceptually easier to work with and require no sophisticated MCMC methods. In this sense, I believe it is a worthwhile addition to the literature. I am generally happy with the paper, with the only note is that the MCMC comparisons are not done particularly carefully, even so, there is enough evidence in favour of the presented method at least on the demo problems. 

questions: In line 256, you say "compute the posterior exactly" but doesn't the adjoint method rely on a truncated basis expansion? 

The adjoints you have used in your models are easy to derive analytically, are there any cases where it is trickier doing this? Are there any models where you can't use the adjoint method and have to use MCMC? I don't have a good understanding of the scope of this new method compared to the old ones.

Why are adjoint solves so fast compared to the forward problem, or at least it seems this way? The use of a single MH step requires two (or one, if the previous one is saved) likelihood evaluations. At the same time in section 4.3 you mention that you did posterior inference for 100 observations, i.e. requiring 100 adjoint solves, in the time it took you to do a single MH step. Why is your MH so inefficient? I understand that the cost of the adjoint solves, once paid, leads to a nice linear system setup and posterior approximation and this may well win over MCMC when the MCMC has to be run for a long time - which is pretty much always. However, I don't think the comparison should be this optimistic.

Finally, you say that M = 10 leads to a poor approximation, but you run your Metropolis sampler with it?

limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper discusses a method that infers the forcing function $f$ of the differential equation $L u = f$ from linear observations of $u$, $z = H u + \epsilon$, $\epsilon \sim N(0, \sigma^2)$.
It assumes a Gaussian process prior over $f$, $f \sim GP(0, k)$, which is approximated with basis functions, $f(x) = \sum_k q_k \phi_k(x)$, for instance random Fourier features.
The essence of the paper is that in the above setup, the observation model can be written as a linear model $z = \Phi q + \epsilon$, where $\Phi = [\langle v_j, \phi_k \rangle]_{i,j}$ is the inner product matrix of the basis elements $(\phi_k)$ and the solutions $(v_j)_j$ of the adjoint equation, $L^* v_j = h_j$, where $h_j$ is the observation functional of the $j$th data point.
This linear model $z = \Phi q + \epsilon$ can be used to infer $q$ from $z$ with standard Gaussian inference tools.

strengths_and_weaknesses: ## Strengths
* The topic is very relevant 
* The methodology is sound and could be useful
* The experiments suggest that the method works
* The related work is more or less covered adequately (with some open questions, see below)


## Weaknesses
While the methodology itself is sound, 
I am struggling with the premise of this work. 
More specifically, I have questions about related work:

* Line 58f states: "All of these approaches require multiple forward solves of the forward problem", but the proposed algorithm also requires multiple forward solves ($n$ solves of the adjoint equation for $n$ data points). For very few data points, this might be cheaper than an MCMC approach, but competing algorithms (see next point) don't seem to require multiple simulations.

* It would be great if the paper discussed why one cannot/should not replace $\Phi q$ with $H L^{-1} f$ for some, say, finite element discretisation of $L$. This seems to be a common approach in the literature on statistical finite element methods; refer to, e.g., Eqs. (21) and (22) in [1].  It is similar to SPDE approaches for Gaussian processes. I think positioning itself against this class of ideas would improve the manuscript. Or have I missed something?

* Section 2 ("Related work") explains how an approach with Green's functions is often not feasible, because Green's functions are difficult to find and potentially unstable to apply. While I don't disagree with this statement, I would appreciate it if the experiments would include a case study in which the Green's function approach cannot be used: the ODE example seems to fit the setup from Alvarez et al.; the advection-diffusion equation appears to also admit a Green's function (see e.g., Eq. 4-3-13 in [2]; however, its representation is quite complicated)




**References**

[1] Girolami, Mark, et al. "The statistical finite element method (statFEM) for coherent synthesis of observation data and model predictions." Computer Methods in Applied Mechanics and Engineering 375 (2021): 113533.


[2] Xu, Zhanjie, Wolfgang Breitung, and J. R. Travis. Green's function method and its application to verification of diffusion models of GASFLOW Code. Karlsruhe, Germany: Forschungszentrum Karlsruhe, 2007.

questions: See "weaknesses" above. 
limitations: -
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is concerned with the inverse problem of estimating the forcing function of f of a linear system Lu=f, with affine linear observations, where the prior on f is a Gaussian process. The authors introduce a trick. They propose to use the adjoint linear system L* v = h in order to rewrite the observation vector as a linear transformation of the parameters of f (plus noise). This of course assumes that f can be decomposed into such an expansion. If f is modeled as a GP, then a truncated basis expansion can play this role - such that the posterior can be inferred.  
This trick has the advantage of turning the model into a standard linear model. Now, standard results can be used to compute least squares (ML) or Bayesian estimators. This is advantageous because these estimators are now only simple linear algebra. The authors derive the trick for Hilbert spaces, but claim that it can be executed for more general Banach spaces.
In the experiments, this approach is spelled out both for ODEs and PDEs. A real time cost analysis is given, comparing with MCMC. Which approach is best, will depend on n. The authors discuss that, if n is not huge, their adjoint approach could outperform MCMC.
strengths_and_weaknesses: Strength: 
1. The paper is easy to read and straightforward. The authors do a good job of highlighting the core of their paper -- how the adjoint trick turns the system into a standard linear model, solvable by linear algebra. The derivation of the adjoint equation reads quite nicely.
2. The literature review is comprehensive.
3. The experiments are instructive and interesting. 
4. The discussion is honest and transparent over the trade-offs in the variable n, compared to MCMC.
5. The method is novel, as far as I know. 
6. The work is significant because (Bayesian) inverse problems are of great interest to the (probabilistic) ML community.

Weaknesses: 
1. A few more experiments could have been given. Right now there is just one ODE and one PDE. More experiments would seem appropriate.
2. The authors compare with none of the methods cited in the Related Work section. It would be great to compare with some of the most recent methods.
3. Comments on the Computational Complexity are spread out over the paper. It would be good to write one section where all of these issues are discussed.
4. In line 152, the authors claim that their trick extends to "more general Banach spaces". I don't understand how. More justification for this claim should be given; perhaps in an Appendix.
5. Section 3.3. is a bit difficult to contextualize. I assume it shows how Bayesian inference of f works, when it is a GP. The crucial bit seems to be that the GP prior can be expressed by a truncated random series, similar to Eq. (8). If so, this connection should be more explicitly made - to improve the clarity of the writing.
questions: 1. Table 1: It is curious that there is no difference between MCMC and Adjoint in this data. Why is that?
2. The linear-parameter assumption of Eq. (8) is common for ODE inverse problems. See e.g. Assumption 1 in https://arxiv.org/pdf/2002.09301.pdf and Eq. (10) in https://proceedings.neurips.cc/paper/2017/file/e71e5cd119bbc5797164fb0cd7fd94a4-Paper.pdf . Maybe comment on that? In any case, it would be good for the reader to know how much of a restriction that assumption entails. What else does the existing literature propose to do with such an assumption?
3. Lines 141-151: Why is the solution to the adjoint equation v_i not part of the ML and Bayesian estimators.
limitations: There are no limitations and potential negative social consequences.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors developed a method to learn unknown forcing functions $f$ of linear systems $Lu=f$ from (affine) observations of $u$, also for PDEs. For finite dimensional learning, this is done symbolically by means of the adjoint. For infinite dimensional learning, a Gaussian process prior is made "finite dimensional" by means of a truncated basis expansion.
strengths_and_weaknesses: **Strengths**
The authors leverage non-trivial ideas from functional analysis to construct better models in a relevant and significant area of research.
The technique is not original in mathematics, but it is a non-trivial extension to machine learning.
Comparison to the state of the art is clearly given (even though, of course it could be longer, but the number of pages is limited).
The language is clear and readable.

**Weaknesses**
1) After reading the paper, there remains a big gap/error in the functional analysis part of the paper. (This might just be a gap/error in the theoretical justification of the method. I do not claim that the method is wrong.)
   Under one viewpoint, three kinds of function spaces appear:
     (i) Hilbert space. They are used a lot, e.g. to compute adjoints. And the realizations of a GP with EQ kernel lie in such spaces (non-trivially, see arXiv:1807.02582).
	 (ii) Banach spaces. I do not know why these spaces appear in the paper.
	 (iii) Frechet spaces, which is the class of spaces your GP-realizations with the EQ kernel have a measure on (see arXiv:2205.03185). (These realizations lie in the space of smooth functions, which is not Banach w.r.t. its usual topology.) 
   Of course, there are Banach spaces suitable for GPs (probably for the Matern kernel, though I do not have a reference).
   Furthermore, the computation of the adjoint as elements in a Hilbert space (using the Riesz representation theorem) depends on the choice of the scalar product. In the example in Section 4.1 such a scalar product is chosen. Hence, the result depends on this choice. This choice (and hence the adjoint, and hence the entire computation) are arbitrary. Even as we have learned in arXiv:1807.02582, there seems to be no canonical RKHS, which contains the GP realizations, and hence no canonical choice of a scalar product. I would assume that the choice of the topology to define the adjoint is clearly justified.
   
   Perhaps I am missing a trivial point here, but I did not find this point in the paper. A summary of the paper according to my feeling is: "Talk about Hilbert and Banach Spaces. Some math magic happens, which we do not explain. This justifies our method." Again, I am not claiming that the method is wrong. But please either explain this math magic or do not try to sound mathmatical.
   If the authors can clarify this point convincingly in a response, I am willing to drastically improve my scores, as otherwise (apart from the following points (2) and (3), which are not too damning) the paper is rather good.
  
2) In my opinion, the authors did not clearly show the benefit of their method experimentally. Neither a comparison with a standard GP was made, nor did the authors compare themselves to related work that might have tackled the same task. Adding this to the experiments would greatly improve the papers value.
3) Experiments on real data would have helped. When dealing with differential equations, synthetic data is usually to "nice".
questions: **Questions**
- I might have failed to understand the explanation for the linear runtime when using a GP and would like the authors to elaborate on this. Is the GP never calculated, but rather $z$ is estimated and using the adjoint, the resulting $f$ can be estimated?
- How general is the suggested method really? Did you try shift operators?
- Is your method applicable to the case when u is not given via observations but as a GP itself? I.e., can you solve linear operator equations for GPs?

**Suggestions**
- A quick reminder in the beginning of 3.1 that $L:\mathcal{U} \to \mathcal{V}$, might improve the flow of reading
- Inconsistencies in how the authors write equations
	- line 172 Eq. 13) and line 179 (Eq. 14) 
- The typesetting of the small brackets in the left part of equation (11) is awkward.
limitations: The authors have not brought up limitations of their work. I do not see a necessity, apart from the above weaknesses.
review_title: null

Meta-review: metareview: The paper looks at a method for inference in Latent Force Models that uses the adjoint method to help out with the inference. Three of the reviewers describe the work as easy to read, sound, relevant and useful. 

Some of the reviewers lament the lack of more experiments - I would have loved to see the results on the spatial experiment described in the discussion. The authors did add one experiment as part of the rebuttal - I urge them to add this to the manuscript. 

One reviewer complained about the 'mathiness' of the presentation. They gave an overall score that I don't feel reflects the overall quality of the paper, and I'm inclined to discard it. Yet, I urge the authors to consider whether all of the terms used are maximising the accessibility and therefore impact of the paper. If Banach spaces are essential to the work, please explain why. If they are a small technical necessity, but unimportant to understand the main ideas, I suggest you relegate them to a formal proof in the appendix. 
confidence: Less certain
Decision: Accept