Title: Inverse M-Kernels for Linear Universal Approximators of Non-Negative Functions
Venue: NeurIPS 2024
Accepted: yes
Keywords: kernel method

m-matrix

non-negative function

intensity estimation

density estimation
Abstract: Kernel methods are widely utilized in machine learning field to learn, from training data, a latent function in a reproducing kernel Hilbert space. It is well known that the approximator thus obtained usually achieves a linear representation, which brings various computational benefits, while maintaining great representation power (i.e., universal approximation). However, when non-negativity constraints are imposed on the function's outputs, the literature usually takes the kernel method-based approximators as offering linear representations at the expense of limited model flexibility or good representation power by allowing for their nonlinear forms. The main contribution of this paper is to derive a sufficient condition for a positive definite kernel so that it may construct flexible and linear approximators of non-negative functions. We call a kernel function that offers these attributes an *inverse M-kernel*; it is reminiscent of the inverse M-matrix. Furthermore, we show that for a one-dimensional input space, universal exponential/Abel kernels are inverse M-kernels and construct linear universal approximators of non-negative functions. To the best of our knowledge, it is the first time that the existence of linear universal approximators of non-negative functions has been elucidated. We confirm the effectiveness of our results by experiments on the problems of non-negativity-constrained regression, density estimation, and intensity estimation. Finally, we discuss issues and perspectives on multi-dimensional input settings.


Peer reviews:
comments: SUMMARY:
The authors combines

- the notion of M-matrix from linear algebra
- the notion of positive semidefinite kernel from kernel learning theory

to create the notion of M-kernel. The M-kernel can be used for solving non-negative constrainted least squares in an RKHS.

STRENGTHS:
- The idea is novel, to my knowledge
- The fusion of these two ideas is an elegant way to efficiently solve the problem of non-negative regression in an RKHS
- The experiments are thorough and demonstrates the theory well

WEAKNESSES:
- There isn't a lot of kernels that are known to be M-kernels (it looks like we only know for 1-dimensional kernels). The authors acknowledge this.
- There are many small grammar/writing errors.

QUESTIONS:
- What is $\gamma$ on L209?
- On L213, is this tridiagonal formula well-known? what is the main proof idea/reference citation to this result?

Typos/word choice/grammar. Number at the front denotes line number.
- 45, "reminiscent" -> "analogue"/"generalization"/"extension" ("reminiscent" isn't used much in mathematical writing to my knowledge)
- 45, "belong" -> "belonging"
- 48 "input space" -> "input spaces"
- 49 "shed the light" -> "shed light"
- 65 "invokes" -> "is given by"
- 71 "which is known as the" -> "a property known as"
- 73 "the computation of O(N 2) and O(N )" -> "O(N 2) and O(N ) kernel evaluations"
- 94 "represents the positive semi-definiteness of matrices" -> "is the positive semi-definite constraint"
- 176 "notorious" -> "well-known" ("Notorious" means "famous for something bad". Unusual for usage in a mathematical context.)

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers learning in RKHS with non-negativity constraints $f(x) \\geq 0$, which turns up with surprising regularity in areas of machine learning.  The proposed approach uses an elegant \"trick\" involving restricting the kernel selection the so-called inverse $M$-kernels that allows the constraint to be enforced in a straight-forward manner (ie adding a simple linear constraint to the optimization).\n\nSTRENGTHS:\nThis paper addresses a problem which I have encountered numerous times in my own research, as such global constraints on GP models seem ubiquitous in applied Bayesian optimization.  I have therefore seen (and used) any number of ad-hoc approaches to dealing with such constraints, typically involving either inducing point grids (which can add significant computational complexity) or latent-space constructs - log-space, function squared etc - that may distort the model in unhelpful and unintuitive ways.  This paper is the first proposal I have seen that appears to offer what I would consider a satisfactory solution to the problem.\n\n- the problem is well-motivated.\n- presentation is mostly readable (though see my concerns below) and the flow of the paper is clear.\n- as far as I can tell the solution is mathematically sound.\n\nWEAKNESSES:\nThe role of the function $s(N)$ needs to be discussed in more depth.  Reading the paper I am left with the impression that $s$ is a bit of a fudge-factor: ideally you would only want to work with strict inverse M-kernels, but in practice they're hard to find so you compromise by including $s$.  My problem here is that you the set of functions $s$ is too broad, including potential arbitrary scaling with $N$.  Speculatively speaking, maybe you could discuss $s$ in big-O terms - you could even categorize inverse M-kernels this way, which would give you a sort of heirarchy (you might argue that a kernel for which $s \\sim O(N)$ suffices is in a sense better than one requiring $s \\sim O(N^2)$).  Alternatively would it be reasonable to borrow from NTK theory (where weight matrices are scaled according to width) and scale the kernel itself by $\\frac{1}{N}$ and replace $s$ with a constant?\n\nI am a little disappointed that the experiments don't include multidimensional examples but it is clear from the discussion that this is a difficult problem so I am willing to overlook this point.\n\nMinor point: please take the time to run this paper through a spell-checker!\n\nQUESTIONS:\nSee weaknesses section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

comments: SUMMARY:
The paper proposes a new kernel family that can be used to fit non-negative functions tractably, which has theoretical novelty. The universal approximation properties are analysed, and a connection is made to permanental processes. The method is applied to univariate regression, density estimation and intensity estimation.

STRENGTHS:
The method contribution has clear originality and novelty. 

The paper is clearly written and presented.

The results are overall good.

WEAKNESSES:
The method is limited to fitting univariate non-negative functions, which is disappointing. Scalar functions can already be handled with existing techniques. This limits the papers significance to low in its current form.

The results are a bit uneven, and sometimes the new method does not seem to fit that well. In Fig1 the QNM is slightly better, in Fig2 the QNM is arguably much better. In FigB2 the new method is much better than baseline, but the fit is still quite poor. Furthermore, all experiments are very simple, and baselines quite simple as well. For instance, there are no GP-based or Cox-process based competing methods. There are also no comparisons to non-negative splines. 

I'm afraid the paper has not been able to find a purpose for their good idea, or show what is the open problem that only it can solve.

QUESTIONS:
None

LIMITATIONS:
No issues
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08