Inverse M-Kernels for Linear Universal Approximators of Non-Negative Functions
Authors: Hideaki Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: kernel method
m-matrix
non-negative function
intensity estimation
density estimation
Primary area: learning_theory

Abstract: Kernel methods are widely utilized in machine learning field to learn, from training data, a latent function in a reproducing kernel Hilbert space. It is well known that the approximator thus obtained usually achieves a linear representation, which brings various computational benefits, while maintaining great representation power (i.e., universal approximation). However, when non-negativity constraints are imposed on the function's outputs, the literature usually takes the kernel method-based approximators as offering linear representations at the expense of limited model flexibility or good representation power by allowing for their nonlinear forms. The main contribution of this paper is to derive a sufficient condition for a positive definite kernel so that it may construct flexible and linear approximators of non-negative functions. We call a kernel function that offers these attributes an *inverse M-kernel*; it is reminiscent of the inverse M-matrix. Furthermore, we show that for a one-dimensional input space, universal exponential/Abel kernels are inverse M-kernels and construct linear universal approximators of non-negative functions. To the best of our knowledge, it is the first time that the existence of linear universal approximators of non-negative functions has been elucidated. We confirm the effectiveness of our results by experiments on the problems of non-negativity-constrained regression, density estimation, and intensity estimation. Finally, we discuss issues and perspectives on multi-dimensional input settings.

Inverse M-Kernels for Linear Universal Approximators of Non-Negative Functions
Hideaki Kim
NTT Corporation hideaki. kin@ntt.com com
Abstract
Kernel methods are widely utilized machine learning field to learn, from train- ing data, latent function in reproducing kernel Hilbert space. It is well known that the approximator thus obtained usually achieves a linear represen- tation, which brings various computational benefits, while maintaining great rep- resentation power (i.e., universal approximation), However, when nog-tegatyv constraints are imposed the function's 's outputs, the literature usually takes the kernel method-based approximators as offering linear representations at the ex- pense of limited model flexibility good representation power by allowing for their nonlinear forms. The main contribution of this paper is to derive a sufficient condition for a positive definite kernel so that it may construct flexible and lin- ear approximators of non-negative functions. We call a kernel function that offers these attributes inverse M-kernel; it is generalization of the inverse M-matrix. Furthermore, we show that for one oee-immensional input space, universal exponen- tial/Abel kernels are inverse M-kernels and construct linear universal approxima- tors of non- non-negative functions. To the best of our knowledge. it is the first time that the existence of linear universal approximators of non-negative functions has been elucidated. We confirm the effectiveness of our results by experiments on the problems of megativvttycconsraaine sity estimation. Finally, we discuss issues and perspectives on multi-dimennional input settings.
regression, density estimation, and inten-
Introduction
Non con-parametic estimation of latent functions continues be of theoretical and practical impor- tance in wide spectrum of disciplines such as signal/image processing [ [2], system control [I2]] geostatistics [B], bioinformaties [28], and clinical research [[]]. Kernel method, one the most established techniques, learns flexible function approximators by embedding data points into higher dimensional reproducing kernel Hilbert spaces (RKHSs) [2R.
2X] For a broad class of lcarning problems. kernel methods invoke the representer theorem [ and recast the infinit-ddimensiona functional problems as their finite-dimensional counterparts, where the obtained approximators have linear representation, i.e., finite linear combinations of kernel functions evaluated data points. Significant computational benefits are attained by the linear representation such as convex optimiza- tion and cheap evaluation/inteeration of approximators.
In recent years, great attention has been paid to kernel methods with non-negativity constraints on function outputs [R, 17, 17, BI]]; crucial applications include non-regaaivity-coostannee regres- sion, density estimation, and intensity estimation. Compared to unconstrained alternatives, non- ereativity-constrimne
kernel methods developed to date faced with problematic trade-off be- tween linearity and flcxibility: the obtained approximators either can have linear representations at the expense of degraded representation power [[Z]], or achieve good representation power (i.e., uni-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

versal approximation) by accepting nonlinear forms [ DIll which incurs substantial computation costs. To the best of knowledge, no non-negativity negativity-constraaned constrained kernel method has been proposed that combines linear representation with good representation power.
In this paper, propose the first linear universal approximator of non-negative functions for dimensional input spaces. First, we derive sufficient condition so that the kernel can construct linear approximator of non-negative function. We call kernel that satisfies this novel condition an inverse M-kernel; it is generalization of inverse M-matrix [8]. Next, we show that exponential/A Abel kernels, which have the universal approximating property [T9. C0J, are inverse M-kernel functions and can construct linear universal approximators of non von-tegative functions for one-dimensional in- put spaces. It is worth noting that the most popular Gaussian kernels do not satisfy the condition de- manded by the inverse M-kernel. Our results shed light on exponential kernels, which have received less attention the literature, as universal kernels for non-negativity-coom constrained approximators on one-dimensional input spaces.
In Section we outline related works and introduce some known results on M-matrix theory used throughout the paper. In Section H. we introduce the inverse M-kernel and construct linear universal approximators of non-negative functions. In Section El we show some important applications of our results, which include aun-negativitt-coosrraaa estimation, and evaluate the effectiveness of our proposal on synthetic dataf. Finally, Section 5 states our conclusions and discuss future works on multi-dimensional input settings.
regression, density estimation, and intensity
2 Background
2.1 Kernel Method-Based Linear Approximator
Let X' be prescribed input space and & ::A. x X + R be positive semi-definite kernel. Then there exists a unique reproducing kernel Hilbert space (RKHS) H4 [26. 29 associated with kernel k(,,)). Given a set of N points {an, E {} and a regularized learning problem:
min JEH2
+ 2(1111
(1)
where L R + is a loss function, and Ω is a non-decreasing function of the squared RKHS norm of f. is well known that the solution of (I) invokes the representer theorem [27 35] and has the representation,
r(() Σ an ank((nn))
For simplicity, the linear regularizer, 2(11)1)) rl1f for r 2 0, is assumed in this paper. The
nimente-dimensionall optimization problem (II can be reduced to a finite-dimensional one in terms of coefficients = (a1,..., .N) R as follows:
min I(Ka) +ra Ko, NERN
K = (e NRN
(3)
Under universal kernels IT 50J] where RKHSs dense in the space of all continuous functions of
X' the linear approximator (2) can approximate any continuous function on X, which is a property known the universal approximation. Each evaluation of the objective function (L) and the linear approximator (2) needs the computation of ((N2) and O(N), respectively.
When non non-negativity constraints imposed target function f 2 0, the literature considers that the linear approximator (2) cannot be applied directly because it generally has negative values, even under non-negative kernel (c.,,) 0. Conventional approaches to address the problem are listed below (Section 2.33)
'Code and data to reproduce the results available http

---

2.2 Linear Approximators with Non-Negativity Constraints
Non-negative coefficients Model (NCM) [| enforces the linear approximator (2) to be non- negative by using non non-negative coefficients and kernel functions,
f-CM(-)= = Σ knnn )) 01 ON 20, k(;,)) ) 0,
(4)
where coefficients a E RN are obtained by solving the optimization problem (E) with constraint On 2 Here non-negative kernels include popular kernels such as Gaussian kernels --1l=~>'1?
exponential kernels e --====111,, and Cauchy kernels (1 (1+1|z - x'1 z²11?) 1 Although NCM enjoys great computational benefits from its linear representation such the preservation of loss functional's convexity and cheap ovaluations/ineeraaionn it suffers from low representation power due to the strong non-negativity constraint on coefficients, On (for details, see Appendix IB).
Note that linear approaches with partial non-negativity constraints [5L, 17, 22],, which require non- negativity only on a finite number of points (the points are not necessarily data points {Ir, J 1 do not guarantee the non-negativity at locations other than the points; these approaches are out of scope of this paper.
2.3 Nonlinear Approximators with Non-Negativity Constraints
An elegant quadratic form of the non-negative model (QNM) [17]] has recently been proposed that exploits the non-negativity inherent in positive semi-definite operators:
[QNM(-)-
Σ k(In, k((a), BERNNN, B 2,n==1
(5)
where > represents the positive semi-definite constraint. QNM has the following beneficial prop- erties: it preserves the convexity of the loss functionals; it can be integrated in closed form if we know how to integrate kernel functions; under mild conditions on kernels, it is a universal ap- proximator for non-negative functions. The coefficient matrix B is obtained efficiently by solving an N-dimensional optimization problem, which naively costs (NN) for each evaluation of the objective function (for details, see Appendix A)). An evaluation of the approximator (5) needs the computation of (
Generalized linear models (GLMs), another nonlinear approach to constructing a non-negative func- tion, use nonlinear transformation sent a wide class of non-negative functions, but they generally do not preserve the convexity of loss functionals where they are used and cannot be integrated in closed form. However, most recently, promising model called squared neural family (SNF) [[44], which was specifically designed for density estimation, has been proposed that uses a quadratic transformation,
linear model [2T]] GLMs
flexible that they can repre-
du(-) =(0) (utt))))]"
fsNF(-)dx
g(t: = Vo + b), Q =V(V. (V, W.b),
(6)
where di(()) is non-negative measure, g( is neural network of hidden layer with tivation function o()) and parameter O t(-) is a sufficient statistic, and x(O) is the normalizing constant, z 2( Jx [lg(t(a); +) [=) Thanks [[MEANING IS UNCLEAR the theory of the neural network, -> ?? Gaussian process kernels [[0]]]] the integrations of SNF (i.e., 2(O) be executed in a closed form under various dp(-), ((-), and (.). ). But SNF has some possible drawbacks: cannot preserve the convexity of the loss functionals, and can yield many local optima; it has
many hyper-parameters to be determined such as (.) and the size parameter θ on which model
performance largely depends. In this paper, adopted the same size of θ in 54 V Rx1330
W R 30x R300 and t(x)
2.4 M-Matrix Theory
Here we introduce some existing results on M-matrix theory, which are then used to clarify a suffi- cient kernel condition so that may construct a linear approximator of non-negative functions.
3

---

The n-by-n real matrix A E R"x" is called an M-matrix [ it has the form yI, C, in which In E RRx is the identity matrix of size n, C -=*= an entry-wise non non-negative matrix, and y> p(C), the spectral radius of C: this is equivalent to A with non-positive off-diagonal entries that is invertible and having an entry-wise non-negative inverse. An entry-wise non-negative matrix that occurs as the inverse of an M-matrix is called an inverse M-matrix [8]. Note that the inverse of an M-matrix is always entry-wise non-negative, while the reverse is not always true. We denote the classes of M-matrices and inverse M-matrices by M and .M-,, respectively. Lemma II below shows useful properties on the entry-wise sign patterns of partitioned inverse M-matrices (for the proof, see Theorem 8 in [S].
Lemma 1. Suppose that an n-by-n symmetric inverse M-matrix Q E EM-
1 is partitioned as
Q Q
Q = Q3
Q Z
Q2 R2 Q3 E R
Q2
where and n2 are positive integers satisfying n1 + n2. Then the following inequalities hold:
0 Q3 Qz
0 Q: - Q Q Q3 E E-
where < represents the entry-wise inequality of matrices/vectors.
Proof. Suppose that the inverse of Q, denoted by Q E EM, is partitioned as
Q Q3
Q =
Q E R
Q2 R 22
Q3 E 22x%
Q3
Q2
Then 11 and Q may be expressed by using Schur's complements as
Q = (Q1 Q Qs) Q = - 00-2--0'0) -z
Because Q1 principal submatrix of M-matrix Q. is an M-matrix (Corollary in [ Q
Q3 Q7 Q3 Q is an inverse M- M-matrix and entry-wise positive. Furthermore, Q3 0₂ - (Q
Q Q₂ Q3 2 is entry-wise positive matrix because (Q1 Q2 Q₂ Q23 M and Q
are both entry-wise positive matrices. This completes the proof.
3 Inverse M-Kernels
In this section, define new class of kernels and show that plays an essential role in constructing a linear and flexible approximator of non-negative functions.
Definition 1 (Inverse M-kernels). Let ousputs non-negative values, K == [K: Fnn) )nn)))n ]an' be gram matrix constructed for any set of N points (x1, F2,-.. ... .NN) ), and 8 N - R2 be a non-negative function of data size N. We call hi k(.,.) an inverse M-kernel if K + IN E M- We denote the class of inverse M-kernels by M-1 Also, if sNN) or K E M-1 we call the kernel a strict inverse M-kernel, which we denote by k(.-) E FM
x - R.+ be a positive semi-definite kernel that
The non-negative function s(N) may generally exhibit various scalings with respect to N, but as will be discussed later, smaller scalings offer greater advantages. In this paper, we will focus solely on examples with scalings of (1)) and ONN).
3.1 Inverse M-Kernel Models
We now consider the following linear approximator with an inverse M-kernel:
(-M = Σ (nnk(nn))) = -(()) (K+ + s(N 1)IN)a o, k(-.) E M
(7)
where (x., N the data points, a a
and k(x) = (k( ..... M(N x)
Coefficients a are obtained by solving the optimization problem (L) with constraint (K + s(N +
1) 2 O. We call this approximator an inverse M-kernel model (IMK). Then Theorem MI below guarantees the non non-negativity of the approximator (.))

---

Theorem 1 (Non-nege -igatttgtt of inverse M-kernel models). The inverse M-kernel models fIM((2) defined by ( are non-negative for any input point x E X.
Proof. Consider the (N++)-by-(NV+1) gram matrix for the data points {In J and any point X such that
k((2, x) k(x)
k(xx) K
Q
s(N + 1)11
Gram matrix Q is an inverse M-matrix because R(;; -) E F and according to the first inequality
in Lemma M1 the following relation holds: k(x) (K + s(N + 1 > 0. Let & E R be non-negative vector,
0, then the following inner product of non-negative vectors completes the
proof: k(x)) KK + s(N - ) = k(x) 2 0, for 3 = (K + s(N + 1) e 0.
should be emphasized here that the comparison between the two linear models, fivk in (7) and JNCM in (E)), suggests that our proposed STıK should have substantially greater representation power than fNCM: SIMK'S s constraint on coefficient, (K + 8 N + 1)xx)a 2 0, much weaker than JNCM ss > O. Here, the discrepancy between the two models is controlled by s(N + +1, and JIMK reduces (s(NN +1) IN) 2 $0000 Clearly, fINK has greatest representation power when s(N 1) is equal to zero, and we can derive sufficient condition on /IMK so that may be a universal approximator of non-negative function.
FNCM for s(N 1) + oo: (K+ I()+)))xxx)
O.
Theorem 2 (Condition for linear universal approximation). The inverse M-kernel model fIMK de- fined by ( is universal approximator of non-negative functions if the kernel is universal and a strict inverse M-kernel.
Proof. Let k X x X - R be universal kernel, and let Z be a compact subset of X. Then the corresponding RKHS equal to the space of all continuous functions from Z, denoted by C(Z), which is equipped with maximum norm lle(z). Suppose that we have a set of data points, {(rn: ((x,)) )) I= for non-negative target function g 2 - R+ in C(2), and k(.,,) is a strict inverse M-kernel k(..-) E FM-1. Then we can rewrite the inverse M-kernel model ([Z) in the form of noise-free kernel ridge regression (KRR) as:
SIMK
(xx)
(.o ((o(=)))..... > 0.
Because constraint g > 0 is satisfied for any {zn) J.N. due to the non-negativity of 9(-), we can apply the generalization error bound normal KRR (Proposition ([S]]) to
()) (()ll((())) sup v M(,, - k(x) K K-ik(w) Г. r
where I is a constant. The upper bound goes to zero if N + xo and { W is aligned appropri- ately, indicating that given >> 0, there exists {.,, ) such that [| (-) g())ll((2) S which completes the proof.
3.2 Equivalent Inverse M-Kernels for Permanental Processes
As by-product of inverse M-kernels, we can address a well-known nodal line problem [ on Pois- son intensity estimation with reproducing kernels [K]] Given set of N points { observed for compact domain T, intensity function A(x), an instantaneous probability of events occurring at each point on T, is estimated by linear model with the equivalent kernel function h(.,.) so that
N
N
A(x) = (xx) f(x) = Σ (a),
e = arg min VERN
log (xx) + F HV, Σ
(8)
mml
where H == [h(2n Fn Naw and f(-..) solves an integral equation constructed by kernel function k(.-)) as h(xxx²)) 2/r ST k(x,,)((s, s) which causes many local modes since + can lead to similar intensity A(-)= - - resulting in artificial zero crossings of f(-), especially on locations where the intensity is low. If i(.,)) is strict inverse M-kernel, then the linear model of f() can constitute an inverse M-kernel model (Z)) which is nen-negative at any E T under weak constraint of coefficient, Hy > 0.
ds = k(x,,'), f(-) generally may have negative values,
5

---

We now consider solving the integral equation for h(-;-) with the naive approach [TO]. which ap- proximates the integral operator by J-point numerical integration, resulting
h(x,z') k(z,z') kz)2)( (wI, + k)) kz())),
w === ==rJ/(2(T)) "k)'),
(9)
where (z) = (k(z, qu).........
9))) K, == [k(q, Iyy |T| = / dz, and (qI..... qJ)
is the regularly aligned evaluation points. Theorem B below shows sufficient condition the equivalent kernel such that it may be a strict inverse M-kernel.
Theorem (Equivalent inverse M-kernels). The equivalent kernel l((.:)) defined by (E) is a strict inverse M-kernel if the corresponding kernel k) is strict inverse M-kernel.
Proof. Consider the (J+2)-by-(J+2) gram matrix for the points { and any pair of different
points (z, # x) E T such that
x(
2(x,2)
k)(
QUU+E, U=
x(2,z)
k(2,2)
ky(2)
E = diag (0, O, w, w,),..,,w)
w).
(z)
kz)())
K,
If k(t))) E FM 1, then U E M- and Q U + E E M because of the additive diagonal
closure of inverse M-matrices (Theorem 3 in [8]). Applying the second inequality Lemma MI to Q leads to the relation: 0 < k(x,z) - (x)) T (wIJ + K)) =kk((=) E FM-1 for any pair of points (I, 2), which completes the proof.
Gaussian Cox processes (GCPs) are the gold standard for intensity estimation, and the intensity estimator (8) is the MAP solution of the permanental process [[8], variant of GCP where the square root of the intensity function is assumed to be generated from a Gaussian process. In the literature GCPs, the advantage of permanental processes over other GCPs has been considered as the efficient estimation algorithm, and the equivalent kernels constructed by inverse M-kernels may improve the predictive performance of the fast-to-compute permanental processes by weakening the coefficient constraints.
3.3 Construction of Inverse M-
In the former sections, defined beneficial results obtained from its unique properties. Now we need to tackle a practical problem of how construct the inverse M-kernels. Our conclusion in this paper is that for one-dimensional input space (X' C IR), we can find some strict inverse M-kernels, which include a well-known universal kernel called axponenial/Abbl//Lalaae kernel; For multi-dimennional input space, we can find some inverse M- M-kernels, but strict ones have yet to be discovered. In the following sections, we focus on the scenario of one- miedimennioona input space, which is followed by discussions of issues with and perspectives on multi-dimmnssonal input setting.
class of kernel or inverse M-kernel, and showed some
Corollary (Examples of strict inverse M-kernels). Exponential kernel kee((z, --lx-=11/ and intersection kernel kmr(z, x') min(æ, x') - 7, defined on one-diment me-dimensional space I, e IR are strict inverse M-kernels. Here, T and y are the hyperparameters of exponential and intersection kernels, respectively.
Proof. Given a set of points (Z1,..., sorted in ascending order xn, < Ine for n < n', the inverse gram matrices of exponential and intersection kernels, denoted by K and Km respectively, are of tridiagonal form:
ph-1
p a++
/pa+1 n+1
n n' = n n- n' =
nnn'' NN otherwise
(K Inn
(vA -1)
I-c 2imn ,//
1
0
n m-n' 1
In pn
In a +1
4m
ml
nn-n' =
2nd -n
: n, n' E NN
(K (Km))an Jan
n''
:
0
In-n'
:n'= = = +1

---

Results on KdV data across 100 trials with standard errors. 1 the integrated squared error between the approximator and the ground truth, and epu is the CPU time in second.
Table I:
NCM
QNM
Our Model
(pu(sec)
12
cpu(sec)
cpu(sec)
0.1
.078
.043
.002 .002
.034 1:
.634 .409
.047
.012
.002 .001
0.01
004 = 0044
.001 = 0011
.022 = 002
3,49 = 1.04
.011 = .001
.003 + .001
1,00
Estimation
NCM
QNM
Our Model
Ground fruth Data paints
0.75
0.50
0.25
0.00
-20
15
-10
5
20 -5
110
20 15
10
-5
5
Figure Estimated non-negative functions on KdV data with small noise, o 0.01.
where Nv 11, NJ........... r O, and 7 < 21- The results show that K=1 and K have non-positive off-diagonal entries while their inverses are entry-wise non-negative due to the cor- responding kernels' properties, indicating that Kexps Kim M for any {xn) Therefore, keeps hint E FM-1.
Because exponential kernel :.) is a universal kernel as well as strict inverse M-kernel, Theo- rem 0 suggests that an inverse M-kernel model with keap(:.)) constitutes a linear universal approx- imator for -ie-imensioonal input spaces. should be emphasized here that more popular kernels such Gaussian and Matérn kernels are not inverse M-kemels, and thus they cannot be used to struct linear universal approximators with non-negativity constraints, which highlights an important benefit of :) :) that has been overlooked in the literature. However, linear models with kepp's)) are generally not smooth at data points, which is a possible disadvantage given that conventional employ smooth kernels. It remains to be clarified whether there exists an inverse M-kernel that is more smooth than (-x)(-.-)
nonlinear models
It is easily verified that linear models with intersection kemnels [06] constitute piece-wise linear splines. By exploiting the fact that piece-wise linear splines, whose finite set of knot values are non- negative, have non-negative values globally, Maatouk and Bay [T4] proposed Gaussian process (or cquivalently a kernel method) model with non-negativity constraints, which our result re-confirms from the perspective of (strict) inverse M-kernels.
We derived the tridiagonal formulae in Corollary M by using some known properties of symmetric Toeplitz matrices (e.g., see [[3]] as a reference. It is clear that the derived tridiagonal formula is straightforward generalization the inverse Toeplitz matrices, but we cannot find any references that explicitly mention the tridiagonal formulae of one-dimensional exponential and intersection kernels.
4 Experiments
We examined the validity of our proposal by comparing it with conventional linear and nonlinear models on synthetic data. Here, we considered the three problems of non-negaaivity-constrimn regression, density estimation, and intensity estimation. As benchmark models, we adopted non- negative coefficients model (NCM) in (e)) and quadratic form of non-negative model (QNM) in regression; we adopted NCM, QNM, and squared neural fam- ily (SNF) in (K) for density estimation; we adopted the intensity estimator with Gaussian kernels (IEK) [R]] and the structured variational Bayesian approach with sigmoidal Gaussian Cox processes (STVB) [II] for intensity estimation. our proposal, we adopted inverse M-kernel model (IMK) in ( for aon--nggatiiityyconttainn regression and density estimation, and intensity estimator with in-
( for non-negativitty.consriaa
7

---

Table 2: Results on density estimation across 100 trials with standard errors. ÍKgL is the Kullback- Leibler distance between the estimation and the ground truth (the lower, the better); cpu is the CPU time in seconds.
NCM
SNF
QNM
Our Model
dKL.
(sc()
dgL.
cpu(scc)
dxL.
cpu(scc)
dxL.
cpu(scc)
.133..154
.024=.007
452+.291
2.99::.970
.163..064
1836.116
.23
.053:t021
NCM
Estimaton Ground trut
SFN
QNM
Our Model
Data mio
0.5
-5
5
-5
-5
Figure 2: Estimated density functions.
verse M-kernels for intensity estimation. We employed Gaussian kernel k(x, x²) ) ---o--1111 for NCM and QNM, and exponential kernel k(x, x²) = e -l-==//// for our IMK. The hyper-parameter for each model were optimized through three-fold cross validation on grid: for NCM, QNM, {0.1, 0.2, 1, 2,5, 10); for SNF, the number of
and IMK, the grid is (T.r) E C @ for components for Gaussian mixture measure du(()) was selected from (1...3). We implemented all compared models by using Python-3. 10.8 (SciPy-1.11. fnnls-1.0 (MIT License)) MacBook Pro with 12-core CPU (Apple M2 Max) was used.
4.1 Non-Negativity-Constaaii Regression
We considered a standard regression problem with the squared loss functional, L 7" En (yn (z,,)) which makes the optimization problems in (Gl and All convex. Here (II,.... .N) 2N) and y (VI,.... )N are the observed input and target values, respectively, and o is the variance of observation noise. For NCM and IMK, each of the convex problems of coefficients a can be recast to non-negative least squares as
NCM min Ca - 2|13 C (KKK/o:vi (TUT) == ly/o;; Onl,
(10)
IMK ming20
| 2112 C -=o VIU-"" = [w/o;;00 a K-13,
where U is the lower triangular matrix of the Cholesky decomposition of the gram matrix, K UUT, Ov is the N-dimensional vector with zero entries, and (a; represents concatenation. We solved (II) with the fast nonnegative least squares [Ø]] For QNM, we solved the convex problem of coefficients B by using the sequential least squares programming (SLSQP) [[]]].
In accordance with [2 we considered approximating a non-negative 2-soliton solution of the Korteweg-de Vries (KdV) equation [25]], 9(2, 1 = 1) 22(3-4440ohh22-- -oohh 644))) where
5[3cush(a- (arff(r--8)))) 28s)) 3361]
the posterior means of unconstrained Gaussian process (equivalently, kernel method regressions) tend to violate non-negativity of the function [22]] We sampled N 40 data points equidistantly from the KdV solution with small noise which was conducted 100 times. Then we measured the predictive performances the models based on the integrated squared error between the result f-) and the ground truth ((x), defined as
0.01 and large noise
0.1 scenarios, each of
= f (f - g(x))² dr for (a, b) = (-2
Table 01 displays the predictive performances achieved by the compared methods on KdV data. The comparison between the two linear models shows that our model (IMK) outperformed NCM in both noise scenarios, which is clearly illustrated by Figure MI our model succeeded in approximating the two modes with the different scales appearing in the KdV solution, while NCM failed due to its limited representation power. See also additional experiment in Appendix B to illustrate the difference in representation power between the two linear models. QNM achieved the best performance among the models, because the underlying function is smooth and consistent with

---

Table 3: Results on intensity estimation across 100 trials with standard errors. 1 the integrated squared enror between the approximator and the ground truth, and epu is the CPU time in seconds.
IEK
STVB
Our Model
(x110))
cpu(sec)
(x110))
cpu(sec)
(2 (xx10))
cpu(sec)
7.50 3.06
6.36 3.46
1.74 + 0.504
1006 + 54.5
2.01 + 0.678
4.53 + 3.15
IEK
Our Model
150
Estimation Ground truth points
100
0
2
3
5
STVB
150
100
50
Figure 3: Estimated intensity functions. Asterisks represent nodal points.
the Gaussian kernel that QNM adopted. The difference in performance between QNM and our model tends to shrink when observations are noisy. The advantage of our method over QNM is its computation efficiency: the linearity of the model can be fully exploited to achieve learning that is hundreds of times faster than QNM.
4.2 Density Estimation
=== We considered a density estimation problem with the loss functional, L - log (I where x = (x1,...,ZN') are the observed samples and the optimization problems in (G) and A convex. Approximators of density functions are required satisfy the normalization condition, which can be recast as a linear constraint the linear models (NCM and IMK): h hTa = for (h), = Sx k(I, vn )dr. We trained NCM, SNF, QNM, and IMK by using SLSQP [I]]].
We created 100 sets of 50 samples generated from a Gaussian mixture model: g(æ) 0.5[A[(=00,
+ (x)44, 0.3)]. where VV-la, b) represents normal distribution with mean a and standard deviation b. The predictive performances were evaluated using the Bullback-Leibler distance (the lower. the better) between the result f*(x) and the ground truth g(z), defined as dxi.
1eth g(x) log(g(x)/ fx))) for () (-.,5). Table Q lists the results, which show that our IMK achieved better performance than NCM and comparable performance while being substan- tially faster than QNM. Table E2 also shows that SNF did not perform well, which might be due to overfitting to the training data, as illustrated by Figure DL This time we assumed small training data set (N 50), where neural setwork-based models are likely to overfit. More careful tuning of the hyperparameters would improve SNF's performances, while the robustness against data size is generally great advantage of kernel methods.
4.3 Intensity Estimation
We considered an intensity estimation problem (8), where SLSQP [[]]] was used to optimize IEK
and our model. We implemented STVB with the TensorFlow code [II], where the number of inducing
points was set as regularly aligned 100 points within the observation domain. We created 100
sets of event sequences generated from the following intensity function: X(x) 50 sin 2 + 60

---

for x E [0,5]] where John and Hensman M reported that the nodal line problem was likely to happen. The predictive performances were evaluated using the integrated squared error between the result X* (a) and the ground truth X(z), defined as 12 = S. [^* >(z) Fdx for (a, b) = (0, 5),
Table lists the results which show that our model with an equivalent inverse M-kernel achieved better performance than the naive intensity estimator with Gaussian kernel (IEK) and comparable
performance to STVB while being substantially faster. Figure B illustrates that IEK allowed some artificial zero erossings of / (x), while our model did not.
5 Discussions
We have proposed novel class of kernel function, called inverse M-kernel function, with which may construct flexible and linear approximators of non-negative functions. We showed that exponential kernels, which are known as universal kernels, are inverse M-kernel functions, and they construct linear universal approximators of non-negative functions for one-dimensional input settings. We confirmed the potential benefits of our proposal experimentally on three problems: non-negative function regression, density estimation, and intensity estimation.
Future Work and Limitations To the best of our knowledge, this study is the first to clarify the ex- istence of linear universal approximators of non-negative functions, although the result is limited to one-dimensional input spaces. Constructing linear and flexible (or universal, if possible) approxima- tors with non-negativity constraints for multi-dimensional input space is equivalent to finding inverse M-kernels for miltt-diimessionll input spaces, the difficulty of which can be exemplified as follows. Let ko IR IR - I be positive semi-definite kernel, and consider the construction of kernel for a two-dimensional input space by a popular multiplicative approach: =') = ko (x, x =o) (y, v/) for (z. v). A gram matrix of k(:,)) :) denoted by K, evaluated over Cartesian grid of input loca- tions, (I1. ... In,) @ (y1, ... Bny ). will give rise to matrix that can written as the Kronecker product of two smaller gram matrices, each of which are formed by evaluating Ao(-.-) over each input location [24]]: K = K2 & K2 y. If ko(:,)) is strict inverse M-kernel function ko E FM-1, then K=- K2 E M, but K
K & K & M,, that is, ((222') & FM 1: For example, (Ky k2t)1²)) 2 0.
some off-diagonal entries of K-1 are non-negative, (K-*) +()
A possible solution to the above difficulty is to select scalar 77 large enough satisfy the condition of inverse M-kernel (Definition I): K + nf E _M For general kernel functions, the condition not always satisfied even under a very large 7). However, if gram matrix K satisfies a specific condition called strict path product condition [9]. then lower bound of n (i.e., s(N) in Definition (1) that satisfies K + nI M-1 can be evaluated as follows.
Theorem (Theorem 4 in [Р). Let == = (Wiz) be an n-by-n entry-wise non-negative matrix with normalized unit diagonals, n 2 3. Then A + E M-1 all n 2 n - 3 if A satisfies the strict path product condition, ai <. 0,k for all distinct indices i, k such that 1 S i, k S n.
Actually, it is easily verified that gram matrices of multiplicative exponential kernels, kesp(1r s) =
Ila -1=""=== satisfy the striet path product condition regardless of the dimensionality of the input space. Therefore, the multiplicative exponential kernels are inverse M-kernels with s(N) N - 3, that is, kexp(-,)) E FN 33 for multi-dimensional input spaces, which suggests that the following inverse M-kernel model (IMK) valid:
fInK(x)= Eo Cak((nn x² = A(xxTo, (K+(N-2) > O, k(x, s)= n=1
II e 1z 11/t,, (11) d=1
where D 2 1 is the input dimensionality. However, as discussed in Section 5.LL. the discrepancy between NCM and IMK becomes small if s(N + 1) for the condition (K + s N + 1)]]aa > 0 is large, and thus s(N +1) N 2 implies that improvements of IMK (II) against NCM (g) should exist but are likely to be marginal for N Z 10. Because the lower bound 2 N 3 in the theorem above is not tight, a pressing need is to develop a method to find a smaller value of 7) satisfying (K +nI) M given kernel k(--)) -) and input points (N)....,
10

---

References
[1] Virginia Aglietti, Edwin V. Bonilla, Theodoros Damoulas, and Sally Cripps. Structured vari- ational inference in continuous Cox process models. In Advances in Neural Information Pro- cessing Systems 32, 2019.
[2] Rasmus Bro and Sijmen De Jong. fast non-negativityyconstii least squares algorithm. Journal of Chemometrics: A Journal of the Chemometrics Society, 11(5):393-411, 1997.
[3] Jean-Paul Chiles and Pierre Delfiner. Geostatistics: Modeling Spatial Uncertainty, volume
713. John Wiley & Sons, 2012.
[4] David Collett. Modelling Survival Data in Medical Research. Chapman and Hall/CRC, 2023.
[5] Sébastien Da Veiga and Amandine Marrel. Gaussian process modeling with inequality con- straints. In Annales de la Faculté des sciences de Toulouse: Matéématiuess, volume 21, pages 529-555, 2012.
[6] Seth Flaxman, Yee Whye Teh, and Dino Sejdinovic. Poisson intensity estimation with repro- ducing kernels. In Artificial Intelligence and Statistics, pages 270-279, PMLR, 2017.
[7] S. T. John and James Hensman. Large-scale Cox process inference using variational Fourier features. In International Conference on Machine Learning, volume 80, pages 2362-2370. PMLR, 2018.
[8] Charles R. Johnson. Inverse M-matrices. Linear Algebra and its Applications, 47:195-216, 1982.
[9] Charles R. Johnson and Ronald L. Smith. Positive, path product, and inverse M-matrices. Linear Algebra and Its Applications, 421((-3):328-33333 337, 2007.
[10] Hideaki Kim, Taichi Asami, and Hiroyuki Toda. Fast Bayesian estimation of point process intensity function of covariates. In Advances in Neural Information Processing Systems 35, 2022.
[11] Dieter Kraft. software package for sequential quadratic programming. Forschungsbericht Deutsche Forschungs- und Versuchsanstali fur Luft- und Raumfahrt, 1988.
[12] Miao Liu, Girish Chowdhary, Bruno Castra Da Silva, Shih- Yuan Liu, and Jonathan P. How. Gaussian processes for learning and control: A tutorial with examples. IEEE Control Systems Maguzine, 38(5):33-86 2018.
[13] Weifeng Liu, Jose C. Principe, and Simon Haykin. Kernel Adaptive Filtering: A Comprehen- sive Introduction. John Wiley & Sons, 2011.
[14] Hassan Maatouk and Xavier Bay. Gaussian process emulators for computer experiments with inequality constraints. Mathematical Geosciences, 49:557-582, 2017.
[15] Emilio Tanowe Maddalena, Paul Scharnhorst, and Colin N. Jones.
Deterministic error bounds for kernel-based learning techniques under bounded noise. Automatica, 134:109896, 2021.
[16] Subhransu Maji, Alexander C. Berg, and Jitendra Malik. Classification using intersection kernel support vector machines is efficient. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8, IEEE, 2008
[17] Ulysse Marteau-Ferey, Francis Bach, and Alessandro Rudi. Non-parametric models for non- negative functions. In Advances in Neural Information Processing Systems 33, 2020.
[18] Peter McCullagh and Jesper Møller. The permanental process. Advances in Applied Probabil- iry, 38(4):873-88) 888, 2006.
[19] Charles A. Micchelli, Yuesheng Xu, and Haizhang Zhang Universal kernels. Journal of Machine Learning Research, 7(12), 2006.
[20] Radford M. Neal. Bayesian Learning for Neural Networks, volume 118. Springer Science & Business Media, 2012.
11

---

[21] John Ashworth Nelder and Robert W.M. Wedderburn. Generalized linear models. Journal of the Royal Statistical Society Series A: Statisties Society, 135(3))370-384, 1972.
[22] Andrew Pensoneault, Xiu Yang, and Xueyu Zhu. Nomnegativity-ennorred Gaussian process regression. Theoretical and Applied Mechanics Letters, 10(3):182-1877 2020.
[23] Robert J. Plemmons. M-matrix characterizations. I-nonsingular M-matrices. Linear Algebra and its Applications, 18(2):175-188, 1977.
[24] Yunus Saatçi. Scalable Inference for Structured Gaussian Process Models. PhD thesis, Uni- versity of Cambridge, 2011.
[25] Nicolas Schalch. The Korteweg-de Vries Equation. Proseminar: Algebra. Topology and Group Theory in Physics, 2018.
[26] Bernhard Scholkopf and Alexander J. Smola. Learning with Kernels: Support Vector Ma- chines, Regularization, Optimization, and Beyond. MIT press, 2018.
[27] Bernhard Schölkopf, Ralf Herbrich, and Alex Smola. A generalized representer theorem. In International Conference Computational Learning Theory, pages 416 Springer, 2001.
[28] Bernhard Schölkopf, Koji Tsuda, and Jean-Philippe Vert. Kernel Methods in Computational Biology. MIT press, 2004.
[29] John Taylor and Nello Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press. 2004.
[30] Bharath K. Sriperumbudur, Kenji Fukumizu, and Gert R.G. Lanckriet. Universality, character- istic kernels and RKHS embedding of measures. Journal of Machine Learning Research, 12 (7), 2011.
[31] Laura P. Swiler, Mamikon Gulian, Ari L. Frankel, Cosmin Safta, and John D. Jakeman. A sur- vey of constrained Gaussian process regression: Approaches and implementation challenges. Journal of Machine Learning for Modeling and Computing, 1(2), 2020.
[32] Hiroyuki Takeda, Sina Farsiu, and Peyman Milanfar. Kernel regression for image processing and reconstruction. IEEE Transactions on Image Processing, 16(2):349-366, 2007.
[33] William F. Trench. Properties of some generalizations of Kac-Murdock-Szeggi Matrices. Con- temporary Mathematics, 281:233-244 2001.
[34] Russell Tsuchida, Cheng Soon Ong, and Dino Sejdinovic. Squared neural families: A new class of tractable density models. In Advances in Neural Information Processing Systems 36, 2023.
[35] Grace Wahba. Spline Models for Observational Data, volume 59, SIAM, 1990.
12

---

NCM
QNM
Our Model
1.00
20
1o100
N=2
N-100
N=220
N
0.75
Ealanelon Ground muth Data points
0.50
0.25
0.00
Figure B1: Estimation results of regression problem on ground truth g(x) = |=|| by using ref- erence models (NCM and QNM) with Gaussian kernel e ///4/11/4 and our model with inverse M-kernel e |a-='1/2
A Details of QNM
Marteau-Ferey al. considered the following problem of positive semi-definite operator A,
inf L...((t)........((.)))) + +r| A21
(A1)
where ||- and 11:113: are the nuclear norm and the squared Frobenius norm, respectively, and fa(+) defined as f-(-) o(-) Ao(-) for o(-) the feature map of A. Then they showed that the solution of (AII holds for representer theorem, which leads to the quadratic form of approximator (5)) The coefficient matrix in (S) obtained efficiently by solving the N-dimensional dual problem,
a' arg sup (a) AERN
ddiag(a) +rr] _
(A2)
2r
= = -V- [Vdiag(a'VV* +rl]_ V-T
(A3)
where [A]- represents the negative part of A (for details, [I)])) V is the Cholesky de- composition of K [k(a, Tnn Nmn'' i.e., K VTV, (a E (a,) represents the Fenchel conjugate of the loss functional L(2 2N) Em= ((zn). More concretely,
I' (an) Un + $ for ((z,) 2 Stn yn)2, and (an) = -(1 + log(-ax)) for
1(z,) log(zn).
B Additional Results
To clearly discern the difference in representation power between the two linear models, NCM and IMK, we conducted experiments on N 20 and 100 data points sampled equidistantly from g(x) = e - with noise a = 0.01, where we set the scale parameter 7 of kernel function to be twice 2, for all models. Figure ВТ displays the results: NCM failed to recover the functional form of the ground truth even with a large number of training points, while our IMK, which invokes the universal approximation under strict inverse M-kernel function, achieved good estimation results. As emphasized by Marteau-Feree et al. [[Z]], NCM cannot approximate function with scale strictly smaller than the used kernel's scale well, which raises problematic trade-off when the underlying function has different scales of components: adjusting to the smaller scale components causes overfitting in the larger scale ones. while adjusting 7 to the larger scale ones fails to recover the smaller scale ones. The experiments in Section 2.1 replicated this problematic situation.
large the ground truth,
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section E is the main contribution of this paper, which is claimed in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are clearly discussed in Sections and
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificatton, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
The theoretical results are numbered and proved in the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the information needed to reproduce the experimental results are Sec- tion EL Also, we will release the code and synthetic data to reproduce the results at github page.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will release the code and synthetic data to reproduce the results at github page. We submitted the code and synthetic data to reviewers.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
/one///uuuddes/oeSumbmmm CodeSubmieSeeecoooc ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the training and test details are in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: All predictive/commpuationnal performances in Section are reported with standard errors.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Information on the computer resources is stated in the first paragraph Sec- tion g
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't tt make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We read the NeurlPS Code of Fthics, and confirmed that our research conform with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is tied to particular applications, and does not present any foreseeable societal consequence.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Synthetic data used in this paper is not tied to particular applications.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We used python package fnnls, of which version and license are mentioned in Section A..
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswitheode www.com//aaaa.c datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
We will also release code to reproduce the dataset used in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "m9n4kRbQPv", "reviewer": "Reviewer_WaA4", "summary": "The paper proposes a new kernel family that can be used to fit non-negative functions tractably, which has theoretical novelty. The universal approximation properties are analysed, and a connection is made to permanental processes. The method is applied to univariate regression, density estimation and intensity estimation.", "questions": "None", "limitations": "No issues", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method contribution has clear originality and novelty. \n\nThe paper is clearly written and presented.\n\nThe results are overall good.", "weaknesses": "The method is limited to fitting univariate non-negative functions, which is disappointing. Scalar functions can already be handled with existing techniques. This limits the papers significance to low in its current form.\n\nThe results are a bit uneven, and sometimes the new method does not seem to fit that well. In Fig1 the QNM is slightly better, in Fig2 the QNM is arguably much better. In FigB2 the new method is much better than baseline, but the fit is still quite poor. Furthermore, all experiments are very simple, and baselines quite simple as well. For instance, there are no GP-based or Cox-process based competing methods. There are also no comparisons to non-negative splines. \n\nI'm afraid the paper has not been able to find a purpose for their good idea, or show what is the open problem that only it can solve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for carefully reading our paper and giving valuable comments. While the reviewer has positive evaluations of the core part of this paper (Soundness: good, Presentation: good, Contribution: good), we understand that the critical concern of the reviewer lies in the ambiguity of the paper's purpose and significance. We can clearly state the purpose and significance of our work, and therefore, we believe that we can fully dispel the concern raised by the reviewer. Below, we provide a detailed response to each of the comments.\n\n**The method is limited to fitting univariate non-negative functions, which is disappointing. Scalar functions can already be handled with existing techniques. This limits the papers significance to low in its current form...** \n**I'm afraid the paper has not been able to find a purpose for their good idea, or show what is the open problem that only it can solve.**\n\nThis paper tackles the *open problem of whether a linear model can simultaneously satisfy non-negativity and universal approximation*. Through the novel concept of inverse M-kernels, we have, for the first time, demonstrated that the answer to the problem is 'Yes' at least in the case of one-dimensional input scenario. While non-linear models that satisfy both non-negativity and universal approximation have been proposed so far, no such linear model has been developed in existing techniques. The linearity of a model offers significant computational advantages in both learning and prediction of latent function, and as Reviewer wAiw kindly commented, its practical value is significant—even in the case of one-dimensional inputs. A comparison of CPU times between QNM (non-linear model) and our linear model in Tables 1 and 2 clearly demonstrates that our model achieved computational speeds tens to hundreds of times faster than QNM.\n\nBased on the reviewer's comments, we found that our explanation of the paper's purpose and significance was inadequate. We will add the above discussion in the introduction.\n\n**In Fig1 the QNM is slightly better, in Fig2 the QNM is arguably much better.**\n\nThis is a misunderstanding by the reviewer, likely due to his/her overlooking that the evaluation metric ($d_{\\text{KL}}$) is the Kullback-Leibler distance (the lower, the better): In Figure 2, QNM is slightly inferior to our model regarding $d_{\\text{KL}}$. We apologize for the unclear explanation. We will add the statement \"the lower, the better\" to the legend and the main text.\n\n**In FigB2 the new method is much better than baseline, but the fit is still quite poor.**\n\nActually, the estimation result achieved by our model in Figure B2 is by no means poor. Event sequence data generated from a point process is inherently noisy, and it is challenging to estimate the intensity function from such data accurately. For example, the center of Figure 2 in (John & Hensman, 2018) shows the estimation result using a variational Bayesian method for the same underlying intensity function, where the result is similar to that of our model in Figure B2. To clarify it, we implemented and evaluated one of the SOTA methods, the structured variational approach with sigmoidal Gaussian Cox processes (Aglietti et al., 2019). Please see the following response about intensity estimation for more details.\n\n**Furthermore, all experiments are very simple, and baselines quite simple as well. For instance, there are no GP-based or Cox-process based competing methods.**\n\n*Regression and Density Estimation*: \n\nIn this paper, we focus on point estimation rather than interval estimation, where GP-based models usually reduce to kernel methods that include our model. An exception is the approach by (Pensoneault et al., 2020), which imposes the non-negativity constraints on finite virtual points by utilizing the confidence intervals of posterior distributions. It is a very different approach from the kernel method that cannot access posterior distributions. However, as mentioned in Section 2.2, the approach is out of the scope of our paper because it does not guarantee non-negativity at locations other than the points. We believe that NCM, QNM, and SNF are appropriate baselines in the literature.\n\n*Intensity Estimation*:\n\nAs the reviewer pointed out, Gaussian Cox processes (GCPs) are the gold standard for intensity estimation. Actually, the intensity estimator defined in Section 3.2 is the MAP solution of the permanental process, a variant of GCP assuming that the square root of the intensity function is generated from a Gaussian process. Therefore, the reviewer's claim that there are no Cox process-based methods in the experiments is inaccurate. However, from the reviewer's comments, we found that other GCP-based methods should be included in the baselines to clarify the contribution of our paper within the context of intensity estimation: In the literature on GCPs, the advantage of permanental processes over other GCPs has been considered as the efficient estimation algorithm, and our work improves the predictive performance of the fast-to-compute permanental processes. We added the estimation result of the structured variational Bayesian approach with sigmoidal Gaussian Cox processes (Aglietti et al., 2019), denoted by STVB. We implemented STVB with the TensorFlow code provided by (Aglietti et al., 2019), where the number of inducing points were set as regularly aligned 100 points within the observation domain. Table C1 and Figure C1 (please see the pdf file in global response) show that our model achieves comparable predictive accuracy while being hundreds of times faster than STVB, which highlights our work's contribution.\n\nJohn and Hensman (2018). Large-scale Cox process inference using variational Fourier features. International Conference on Machine Learning.\n\nAglietti et al. (2019). Structured variational inference in continuous cox process models. Advances in Neural Information Processing Systems 32.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P9fOraFLY4", "reviewer": "Reviewer_wAiw", "summary": "The paper considers learning in RKHS with non-negativity constraints $f(x) \\geq 0$, which turns up with surprising regularity in areas of machine learning.  The proposed approach uses an elegant \"trick\" involving restricting the kernel selection the so-called inverse $M$-kernels that allows the constraint to be enforced in a straight-forward manner (ie adding a simple linear constraint to the optimization).", "questions": "See weaknesses section.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "This paper addresses a problem which I have encountered numerous times in my own research, as such global constraints on GP models seem ubiquitous in applied Bayesian optimization.  I have therefore seen (and used) any number of ad-hoc approaches to dealing with such constraints, typically involving either inducing point grids (which can add significant computational complexity) or latent-space constructs - log-space, function squared etc - that may distort the model in unhelpful and unintuitive ways.  This paper is the first proposal I have seen that appears to offer what I would consider a satisfactory solution to the problem.\n\n- the problem is well-motivated.\n- presentation is mostly readable (though see my concerns below) and the flow of the paper is clear.\n- as far as I can tell the solution is mathematically sound.", "weaknesses": "The role of the function $s(N)$ needs to be discussed in more depth.  Reading the paper I am left with the impression that $s$ is a bit of a fudge-factor: ideally you would only want to work with strict inverse M-kernels, but in practice they're hard to find so you compromise by including $s$.  My problem here is that you the set of functions $s$ is too broad, including potential arbitrary scaling with $N$.  Speculatively speaking, maybe you could discuss $s$ in big-O terms - you could even categorize inverse M-kernels this way, which would give you a sort of heirarchy (you might argue that a kernel for which $s \\sim O(N)$ suffices is in a sense better than one requiring $s \\sim O(N^2)$).  Alternatively would it be reasonable to borrow from NTK theory (where weight matrices are scaled according to width) and scale the kernel itself by $\\frac{1}{N}$ and replace $s$ with a constant?\n\nI am a little disappointed that the experiments don't include multidimensional examples but it is clear from the discussion that this is a difficult problem so I am willing to overlook this point.\n\nMinor point: please take the time to run this paper through a spell-checker!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the highly positive and constructive comments. It is incredibly encouraging to know that a reviewer with practical experience in the estimation of non-negative functions finds value in our work! Below, we provide a detailed response to each of the comments.\n\n**The role of the function needs to be discussed in more depth...**\n\nThank the reviewer for the pertinent comments. We agree with the reviewer about the role of $s(N)$ being ambiguous in the paper. According to the reviewer's suggestion, we will introduce $s$ as a non-negative factor at the definition of inverse M-kernels, and discuss a possible scaling of $s$ with data size $N$: The ideal case is $s = 0$ (strict inverse M-kernels), where inverse M-kernel model $f_{\\text{IMK}}$ has greatest representation power, but we can only find such kernels on one-dimensional input scenario; By exploiting the strict path product condition (for details, see Future Work and Limitations), we can find inverse M-kernels with $s \\sim \\mathcal{O}(N)$ regardless of the input dimensionality, where $f_{\\text{IMK}}$ has limited representation power for a large $N$; An essential next step is to investigate a more scalable factor (e.g., $s \\sim \\mathcal{O}(N^{1/2})$ ) according to observed data, which could relax the limited representation power of $f_{\\text{IMK}}$ in multi-dimensional input settings.\n\n**please take the time to run this paper through a spell-checker!**\n\nSorry for the inconvenience. We promise to improve the grammar quality of the paper by using English proofreading services. We appreciate the reviewer's efforts to read our paper despite the poor English.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5bR4TRYCzz", "reviewer": "Reviewer_dTaH", "summary": "The authors combines\n\n- the notion of M-matrix from linear algebra\n- the notion of positive semidefinite kernel from kernel learning theory\n\nto create the notion of M-kernel. The M-kernel can be used for solving non-negative constrainted least squares in an RKHS.", "questions": "- What is $\\gamma$ on L209?\n- On L213, is this tridiagonal formula well-known? what is the main proof idea/reference citation to this result?\n\nTypos/word choice/grammar. Number at the front denotes line number.\n- 45, \"reminiscent\" -> \"analogue\"/\"generalization\"/\"extension\" (\"reminiscent\" isn't used much in mathematical writing to my knowledge)\n- 45, \"belong\" -> \"belonging\"\n- 48 \"input space\" -> \"input spaces\"\n- 49 \"shed the light\" -> \"shed light\"\n- 65 \"invokes\" -> \"is given by\"\n- 71 \"which is known as the\" -> \"a property known as\"\n- 73 \"the computation of O(N 2) and O(N )\" -> \"O(N 2) and O(N ) kernel evaluations\"\n- 94 \"represents the positive semi-definiteness of matrices\" -> \"is the positive semi-definite constraint\"\n- 176 \"notorious\" -> \"well-known\" (\"Notorious\" means \"famous for something bad\". Unusual for usage in a mathematical context.)", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- The idea is novel, to my knowledge\n- The fusion of these two ideas is an elegant way to efficiently solve the problem of non-negative regression in an RKHS\n- The experiments are thorough and demonstrates the theory well", "weaknesses": "- There isn't a lot of kernels that are known to be M-kernels (it looks like we only know for 1-dimensional kernels). The authors acknowledge this.\n- There are many small grammar/writing errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the highly positive comments, by which we are strongly encouraged! We feel sorry for the grammatical flaws in this paper. Despite this, we appreciate your thorough understanding of our work's content and value. We promise to improve the grammatical quality of this paper using a professional English proofreading service. Below we provide a detailed response to each of the comments.\n\n**There isn't a lot of kernels that are known to be M-kernels (it looks like we only know for 1-dimensional kernels).**\n\nAs the reviewer pointed out, this paper succeeded in only showing two examples of strict inverse M-kernels, which could be a limitation of this paper. However, we are optimistic about the future. This paper successfully, for the first time, clarified a novel condition that allows a linear model to achieve both non-negativity and good representation power, that is, the condition of strict inverse M-kernels. By presenting the condition to a high-level NeurIPS audience, we believe that the open question of how to construct strict inverse M-kernels in a more general manner would be addressed eventually.\n\n**There are many small grammar/writing errors**\n\nWe apologize for the inconvenience and appreciate the kind suggestions about writing, which we will reflect in our paper. We will ensure that the writing quality of the paper is improved through professional proofreading.\n\n**What is $\\gamma$ on L209?**\n\n$\\gamma$ on L209 is the hyperparameter of intersection kernel $k_{\\text{int}}(x,x')$, which should be determined based on data. We will ensure that it is clearly stated in the paper. If $\\gamma$ on L339 confused the reviewer, we apologize for it: $\\gamma$ on L339 is the typo of $s(N+1)$.\n\n**On L213, is this tridiagonal formula well-known?**\n\nWe derived the tridiagonal formula by using some known properties of symmetric Toeplitz matrices (e.g., see Trench 2001) as a reference. It is clear that the derived tridiagonal formula is a straightforward generalization of the inverse of Toeplitz matrices, but we cannot find any references that explicitly mention the tridiagonal formulae of one-dimensional exponential and intersection kernels. We will add the above discussion to the paper.\n\nTrench, W. F. (2001). Properties of some generalizations of Kac-Murdock-Szego matrices. Contemporary Mathematics, 281, 233-246.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their valuable comments. We responded in as much detail as possible. We believe that we can entirely dispel the concerns raised by the reviewers. Please see the added figure (Figure C1) in response to Reviewer WaA4's suggestion.

Meta-review: This paper presents an interesting new method of constructing sufficient conditions on positive definite kernel so that it may construct flexible and linear approximators of non-negative functions.  It is shown connected to the permanental process.  Experimental results demonstrate that the proposed method is competitive.

Overall, the method is novel and the performance on 1-D input is encouraging.   The linear universal approximation property is also valuable. However, the construction of inverse M-kernel is still open for multi-dimensional input, and no experimental results are shown for it.  This considerably limits the significance and applicability of the method.  That said, balancing the overall merit, I’m inclined to recommend accepting this paper.
Decision: Accept (poster)