Talk:Reproducing kernel Hilbert space

message: The first part of this section is very disconnected from the last sentence which reads
"Using this formulation, we can apply the Representer theorem to the RKHS, letting one prove the optimality of using ReLU activations in neural network settings. "
The first part of the section just shows that the ReLu is a reproducing kernel when a Hilber space is considered. 
The Representer theorem shows that when optimizing a sum of two functionals, one just depending on a few points of a function and the second being monotonic on the norm of its input, that we only really need to optimize
the objective on the function on the points that the first functional depends on.
When one rights "letting one prove the optimally of using Relu activation' what kind of optimallity is the text referring too? Is there some citation to go with this ? Typically, when using NN, one does not optimize the activaiton functions, but the weights.
How is the Representer theorem then used in this context?  — Preceding unsigned comment added by 136.167.52.51 (talk) 21:39, 11 July 2020 (UTC)
thread_title:  Problem with section on connection between RKHS and ReLu and Neural Networks 

message: What is this sesquilinearity convention of which you speak? Do you mean linear in the second variable and conjucate-linear in the first? If so, it seems a strange convention to me. Lupin 08:39, 6 Aug 2004 (UTC)
thread_title:  This sesquilinearity convention 

message: The article states that H is a reproducing kernel Hilbert space iff the linear map f→f(x) is norm-continuous for any element x of X. -- what does norm-continuous mean in this context? linas 15:06, 25 Jun 2005 (UTC)
replies: text:  H is a Hilbert space, thus carries a norm. This f → f(x) has a well-defined topology on both source and target.--CSTAR 15:38, 25 Jun 2005 (UTC)
thread_title: Norm-continuous

message: Some examples would be handy too. Reading this article naively, one has the Dirac delta function playing the role of K, so that 
f(x)=\int \delta(x-y) f(y) dy and δ is usually naively understood to be the "identity operator" on the Hilbert space in question (that is, its the sum over all states, e.g. the sum over all orthogonal polynomials on an interval).  Is this naive reading correct? Can I correctly say that an example is
:K(x,y)=\sum_{n=0}^\infty T_n(x) T_n(y)  
where T_n ar the Chebyshev polynomials (and the norm/volume elt/weight is taken appropriately)?  It seems to fulfill the requirements described in this article, I think ... linas 15:21, 25 Jun 2005 (UTC)
replies: text:  That's just it, the delta function is not in the space.--CSTAR 15:38, 25 Jun 2005 (UTC)
replies: text: Would you object if I added the above as an example to the article? linas 18:31, 25 Jun 2005 (UTC)
thread_title: Examples, too

message: The most recent edit claims that the inner product given in the proof of Moore-Aronszajn may be degenerate but doesn't say why. I think that it cannot be degenerate: suppose it is. Then there exists a point x such that \langle K_x, K_x \rangle = 0 and Kx is not the zero function. Then we can find y such that Kx(y) is not zero. Then the kernel matrix determined by x and y is
:
\begin{bmatrix}
0 & a \\
\overline a & b \\
\end{bmatrix}
for some non-zero a and some real number b. One can check that the eigenvalues are 
:
\frac{b \pm \sqrt {b^2 + 4|a|^2}}{2}
and so one eigenvalue is negative, contradicting the fact that the kernel was positive definite. Unless someone can poke a hole in my argument, I'll reinstate the claim that the construction produces a genuine Hilbert space. Jneem 12:14, 27 September 2007 (UTC)
replies: text: I see no flaw in your argument. I'll undo my edit. Memming 13:08, 27 September 2007 (UTC)
replies: text: You were right to require symmetry of the kernel, though, so I'll put that back in. Jneem 00:17, 28 September 2007 (UTC)
thread_title:  It <em>is</em> a Hilbert space 

message: Should we be following the convention for inner product spaces of \langle \cdot , \ \cdot \rangle being linear in the first variable, or the convention for more general sesquilinear forms of being linear in the second variable? My feeling is that since we're dealing solely with Hilbert spaces here, so we're only dealing with inner products, we should use the inner product space convention, which also seems to be more common in the literature. What say you? James pic 11:35, 18 October 2007 (UTC)
replies: text: matter of consensus. i personally don't have a preference. article just needs to be consistent. Mct mht 19:19, 18 October 2007 (UTC)

text: I don't mind terribly -- I used linearity in the second variable for consistency with the page on sesquilinear forms (the page on inner product spaces is inconsistent: first, it says that an inner product is a sesquilinear form. Then it says that an inner product is linear in the first variable. Then it gives an example of an inner product (the third example) that is linear in the second variable). Anyway, if you want to make it linear in the first variable, you should also change the proof of the Moore-Aronszajn theorem. Jneem 01:47, 20 October 2007 (UTC)
replies: text: Like I said, my preference is to use linearity in the first variable, as this seems to be most consistent with the literature, at least for Hilbert spaces. However, even this could be contentious, as physics literature generally uses the opposite convention. As you said, it should be a matter of consensus, so I thought I'd call for a vote.
First Variable: I've seen it most often in the literature. James pic 16:20, 22 October 2007 (UTC)

text: looks like there won't be any objection if you wanna switch conventions. Mct mht 20:38, 24 October 2007 (UTC)
thread_title: Sesquilinearity

message: I do not think that your proof on the fact that given a kernel K, there exists a unique RKHS associared with it is correct. Or, at leats, it is incomplete. After introducing G as another Hilbert space of functions for which K is a reproducing kernel, to prove uniqueness it is assumed that G is the completion of the span of \{K_x:\ x \in E \}. But where does this fact come from? This has to be proved. Notice that in Cucker and Smale's paper "On the mathematical foundations of learning" this is a condition which is listed in the statement of theorem 2 on pag. 35.  —Preceding unsigned comment added by Spline (talk • contribs) 17:34, 10 December 2008 (UTC)
replies: text: I understand that H (or G as well) is a space of functions on E: if an element f ∈ H is orthogonal to all the \{K_x:\ x \in E \}, then f is the 0 function on E, thus f = 0_H, hence \{K_x:\ x \in E \} is total. Does it make sense? Bdmy (talk) 11:38, 11 December 2008 (UTC)
thread_title:  Is the proof correct? 

message: I find the wording "the kernel K is symmetric" ambiguous or misleading. Clearly what is meant but never said is the "Hermitian symmetry"
:K(y, x) = \overline {K(x, y)}
like the one for the Szegö kernel. I am right? Bdmy (talk) 11:43, 11 December 2008 (UTC)
thread_title:  Symmetric or "Hermitian symmetric" 

message: Subsuming large parts of the discussion above, it seems that the article should be  more precise and clearer right at the beginning about the fact that usual L^2-spaces on manifolds are not RKHS's. These aren't because they are not defined as Hilbert spaces of functions, but only of equivalence classes of functions; leading to evaluation at a point of the manifold being undefined. This point is just too likely to be confusing for readers unfamiliar with the subject but used to the usual L^2-spaces.
Wurzel33 (talk) 19:36, 1 December 2013 (UTC)
replies: text: I agree, made this point in the first paragraph, hope this reads well. Biker333 (talk) 11:27, 21 February 2016 (UTC)
thread_title:  Contrast with L^2-spaces 

message: On the page Vashishtha Narayan Singh I have linked his PhD subject "Reproducing Kernels and Operators with a Cyclic Vector" to this page, but I am not sure if this is correct. Could someone please check this? Further, on Wiki Data he is described as "mathematician known for challenging Einstein Gauss theory". Is this correct too? If so, could this be added in the text as well, if possible with an explanation what this implies? I have no knowledge about this subject. Wiki-uk (talk) 11:53, 12 June 2015 (UTC)
thread_title:  Reproducing Kernels and Operators with a Cyclic Vector (V.N. Singh) 

message: The examples are mostly a list of kernels, but this doesn't give a clue as to which functions are in the corresponding RKHS, and what the norm is. I added a sentence to partially complete this for the linear and Laplacian kernels, do others know more, e.g., is anything known about the double exponential case? Biker333 (talk) 00:26, 20 February 2016 (UTC)
thread_title:  Expanding the examples 

message: Let {\displaystyle X}X be an arbitrary set and {\displaystyle H}H a Hilbert space of real-valued functions on {\displaystyle X}X, equipped with pointwise addition and pointwise scalar multiplication.
Hilbert space which is also a vector space???
Then further down the Hilbert space can be given an inner product. I'd suggest reviewing the definition of a Hilbert space before yammering on about RKHS.
A banach space with an inner product. Izmirlig (talk) 02:47, 5 July 2022 (UTC)
replies: text: I think the point is that H is a Hilbert space of functions and not just some abstract vectors, because a function can be evaluated at a point, while a vector (i.e. an element of a linear space) does not have this property. One related remark is made in the article: L^2 is not a space of functions (but a space of classes of equivalence). Also,  the discussion from 2005--2007 just above touches upon how the "evaluation at a point" property is not preserved under isomorphisms between Hilbert spaces, which (if my understanding is correct) is precisely because "evaluation at a point" is not in the structure of a Hilbert space. Thus, being specific that we consider Hilbert spaces of functions is not only correct but also necessary. AVM2019 (talk) 21:54, 9 September 2022 (UTC)
thread_title:  This is a mess. For example,  the first sentence of the definition, 

message: Parts of the article refer extensively to a single source, Lorenzo Rosasco's MIT course 9.520. This includes references to:
*Durrett, Greg.  9.520 Course Notes, Massachusetts Institute of Technology, https://www.mit.edu/~9.520/scribe-notes/class03_gdurett.pdf, February 2010.
* Rosasco, Lorenzo and Poggio, Thomas.  "A Regularization Tour of Machine Learning – MIT 9.520 Lecture Notes" Manuscript, Dec. 2014.
The latter is not publicly available (neither free nor for a fee). The former are lecture notes for one of the classes apparently put together by one of Prof Rosasco's students (and has misattributed authorship).
I suggest to remove these references and replace them by more appropriate primary sources, such as endorsed books or peer-reviewed articles.
(Incidentally, on the course page, Prof. Rosasco encourages students to contribute to Wikipedia as part of assignments, which being a commendable thing, may explain some peculiarities of this article) AVM2019 (talk) 15:48, 10 September 2022 (UTC)
thread_title:  References and citations: more primary, less secondary 

message: Is the function f(x, y) = e^(x, y) a valid kernel function?
While there are some examples of kernels, would you like to expand upon the examples list? Inlin (talk) 11:26, 9 July 2023 (UTC)
thread_title:  Kernel Examples and Kernel Type Query