Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies a method of solving functional inverse problems. The problem set-up is (X,Y) are from a particular distribution, with Y = A[f](X) + noise, where f is some function defining the relationship between X and Y. The major example is: if X is a function on [0,1], then Y = int_0^1 f(t)X(t)dt + noise. The goal of the general problem is to find f.

The paper introduces a means of finding an appropriate f from a class of functions by stochastic gradient descent, emphasizing results for a particular context: functional linear regression. They review previous results and other techniques for solving stochastic inverse problems, define the problem rigorously, and present example problems. They then give the main algorithm, a sort of functional SGD, and identify issues with it due to functional complexity involving the kernel. To address this, they provide another algorithm which uses the iteratively improving f in the SGD algorithm to teach common ML learners, which are used to build f instead of a kernel. They provide probabilistic error bounds and show a probabilistic convergence rate of O(1/ \sqrt(n)). They then present synthetic and natural experiments, with a comparison to another method.
strengths_and_weaknesses: Strengths:
-great organization
-clear examples
-good mathematical exposition
-solid theoretical results
-clear writing
-interesting and good experimental results
-overall interesting problem

weaknesses:
- experimental results seem inconsistently or incompletely reported.
- a few vagaries in mathematics (see questions)
questions: Questions:
- On Theorem 4.5, you claim the result is independent of the dimensionality of the codomain of L_2(X) and L_2(W), the values of n and k in R^d and R^k. Your proof deals with d=1. Are you sure your results are independent of the value of d? If you are sure, perhaps you could include a comment about how all inferences are independent of the values of k and d?
- Table 4 says SGD outperforms FPLR, but figure 3 indicates FPLR outperforms SGD. Is this an error? Furthermore, I find it odd that all FPLR have the same error in table 4. Are they the same function? (Aside: is SGD = SGD-SIP?)  This makes it look like some of your experimental results are incomplete or incorrect. Furthermore, the thoroughness of the deconvolution results are much less thorough than the FLR in the appendix. Please comment.
- The order of convergence for the algorithm, O(1 / \sqrt(n)), seems not so great, and the algorithm that has this guarantee looks jagged and compares poorly to the already established method. Can you comment on the order of convergence, and how it compares to other methods?

Minor issues:
- Can you provide a table for results of B2?
- Is the equation at the bottom of page 3 a component-wise integral, since X takes values in R^d but f takes values in R? This may trouble readers if not mentioned. If this is true, is it necessary X take values in R^d?
- On (138). If x is a specific realization of X, then I believe your conditioning is backwards: you want \phi(x) = E(\partial*l(Y,A[f](X))| X = x). Or do you write it the way you do because you use eq (1), with Y still a random variable of the noise, although is X specified? If this is the case, are you taking the expectation only over the noise, \epsilon? On the other hand, if x is set and Y takes one value for a specified x, what distribution is the expectation over?
- Dimensionally not accounted for in (117), (141), etc. EG: A(g)[X] in R^d, so (A[g](X))^2 not defined if d>1). Also, \partial l(...)*A[g](X) is R^d x R^d, should you should use transpose or standard inner product?
- Posed assuming the 1-d case, dropping the inner-product brackets (144,147,...). Not sure if this is notationally standard.
- (147) A*[g](w) is in R^k, but E(\Phi((X;w)g(X)) is in R^d. Even if d = 1, dimensions do not match. Is \Phi in normally in R^{k x d}?

Typos/minor issues:
- Sentence in (64-66) is confusing, hard to understand.
- "a prespecified basis functions" (74-75)
- "to control directly to tackle (3) directly" (103-104)
- "Y is giving by" (bottom of page 3)
- "also in deconvolutional problems" (112)
- indeed have a solution (128)
- does not dependent (136)
- a coarser grid where and each (214-215)
- Broken references in (380)
- Figure 3 says FPCR.
limitations: - This is my first time being exposed to the topic of the statistical inverse problem, so I am unfamiliar with some of the relevant literature.
They apply the method to standard quadratic loss, and only to FLR and deconvolution. I would like to see an exposition, even if brief, supporting the potential breadth of application. Even better, applications to more problems. This may due to my lack of familiarity with the subject.
- However, a low rate of convergence for the excess risk, and some choppy graphs for algorithm 1 seem a liability, To be fair, this is made up for with strong and interesting theoretical results that could be built of of, and a strong showing for the ML-assisted algorithm.
- Is it possible the ML-assist may not give much of a boost, or may do harm, if the kernel-sampling in not well suited to the ML-learners?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose an SGD approach to solve statistical inverse problems (SIPs). Formally, the problem is to find the parameter $f$ satisfying the relation $Y = A\[f\](X) + \varepsilon$, where the operator $A$ is known, and $Y$ and $X$ are given observations. This can be achieved by minimizing the quantity $\mathcal{R}_A(f) = E\[ l(Y, A\[f \](X))\]$, where $l$ is a loss function. For $A$ linear and bounded, and with additional (milder) assumptions, the authors are able to write the gradient of $\mathcal{R}_A(f)$ as the average of a function $u_f$, whose expression has to be derived explicitly for the specific SIP considered. With this result at hand, the authors propose two algorithms to solve SIPs in the spirit of SGD, where the average giving the gradient is approximated by one sample at each iteration. Their main result proves a performance guarantee for such algorithms. The authors apply the proposed techniques to functional linear regressions, performing numerical tests both on synthetic and real data, and deconvolution (discussed in the supplementary material).
strengths_and_weaknesses: The paper proposes an SGD approach to solve SIPs, thus providing, to the best of my knowledge, an original contribution (at least within the probabilistic formulation of inverse problems). The exposition is clear and compact (section 5 could be improved, though), and the related literature is well covered. 

My main concerns with this work are the following:

1. General applicability of the proposed method. Even considering the (not so week) Assumption 4.1, the proposed method crucially depends on the possibility of identifying the kernel $\phi$ appearing in Lemma 4.3. The authors do not discuss the limitations associated with identifying $\phi$, and only present applications where this is doable. Moreover, the proposed algorithms are designed (or, at least, presented) following Corollary 4.4, i.e. considering the squared loss function. It is unclear to me if Theorem 4.5 applies only under this corollary. 

2. Actual improvements over other methods. The comparisons proposed by the authors seem to suggest that their algorithms are at most as good as the FPLR (see Table 2). I wonder then what are the actual advantages of solving SIPs with the proposed SGD algorithms, particularly considering the previous point. 
questions: I have the following questions/comments for the authors:

1. I would appreciate it if the authors could comment on my concerns in the previous section. I believe that addressing these points in the main text would be beneficial for the paper.

2. The presentation of numerical results could be improved. I believe that Figures 1 and 2 in section 5.2 are a little aimless, as they do not really support any claim in the paper (they take up quite some space, though). Moreover, the label font size in Figure 1 is too small and the x-axis has no scale (or ticks). It is unclear what's on the y-axis in Figure 2 ($f(s)$? If so, we don't know the real $f$ anyway...what is the point?). In any case, I would suggest combining the two figures and using the same format for both. The synthetic data experiments could be useful to test robustness against the signal-to-noise ratio.

3. The sentence 'We highlight that those choices of splines and penalty term are widely used in the literature', line 230, might need a reference.

4. I am not sure I understand what 'with only one observation' means in this sentence: 'The main benefit is that with only one observation we are able to compute an unbiased estimator for the gradient of the risk function under the true distribution', line 158/159. Can you clarify?

5. I would change the article 'An' in the title with 'A'.

6. Line 215 'where and each' -> 'where each'?

7. Line 103/104 'to control directly to tackle (3) directly' -> 'to tackle (3) directly'?
limitations: The authors have partially addressed the limitations of their work, though there is space for improvement (see the section Strengths And Weaknesses).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper mainly focuses on the formulation of SIP (statistical inverse problem) and the method to solve SIP.
In the first part of this paper, the authors formulate SIP utilizing functional analysis. They also give some examples of SIP under this formulation.

In the second part, after showing how to calculate the gradient under this setting, they propose two SGD-based algorithm to solve SIP. They give a finite sample bound for the expected excess risk of their algorithms.

In the last part, the authors provide numerical results for two applications of the Functional Linear Regression problem. 
strengths_and_weaknesses: Strengths: The authors give a novel numerical method to solve SIP, which shows good originality. They give both theoretical analysis and real data experiments, which shows the good quality of this paper. 

Weakness: It seems that they do not show the intuition and advantage of their new method, which makes this paper somehow unclear. And the proof techniques are just classical convex optimization analysis. Therefore this paper lacks significance to some extent.
questions: 1.In Lemma 4.3, you've mentioned that there exists a kernel \Phi. Can this kernel be easily found in many SIP problems (other than functional linear regression)? 


limitations: It might be interesting if the results can be extended to nonlinear SIP (or any other relaxation of Assumption 4.1).
review_title: null

Meta-review: metareview: All reviewers recommend accepting the paper. Congratulations!

But in your camera-ready version, please revise the paper to better emphasize the relevance of this line work to a general machine learning audience. For example, during the discussion one reviewer wrote the following:

"The authors have not thoroughly convinced me of the relevance to a ML audience, as I will explain. To me, the relevance is very likely, as they apply SGD to a regression problem, and then ML learners on top of that, which is very much in the vein of ML. My confidence is limited, however, since I am not familiar with functional regression.

The paper could better appeal to an ML audience by mentioning ML applications of their method (whether it be functional linear regression or otherwise) early in their paper and in an appealing way. Section 5.2 (Real Data Application) does a good job of this, but giving motivating use cases could help pique reader interest/help them understand application earlier."

The paper will have greater influence if the final version can convince readers of its relevance to ML!
confidence: Certain
Decision: Accept