Functional learning through kernel

This paper review the functional aspects of the statistical learning theory. It regards mainly the nature of the hypothesis set when no prior information is available but the data. In this framework our starting point is a discussion arguing three principles about the hyp othesis set: it is a vectorial space, it is a set of pointwise defined functions and the evaluation functiona l on this set is a continuous mapping. Based on these principles an original theory is developed general izing the notion of reproduction kernel Hilbert space to non hilbertian sets. Then it is shown that the hypoth esis set of any learning machine has to be a generalized reproducing set. Therefore, thanks to a gener al “r presenter theorem”, the solution of the learning problem is still a linear combination of some kerne l. Furthermore, a way to design these kernels is given. To illustrates this framework some examples of suc h reproducing sets and kernels are given. 1 Some questions reading machine learning Kernels and in particular Mercer or reproducing kernels pla y a crucial role in statistical learning theory and functional estimation. But very little is known about th e associated hypothesis set, the underlying functional space where learning machines are looking for th e solution. How to choose it? How to build it? What is its relationship with regularization? The machi ne learning community has been interested by the over way round. For a given learning task, thus for a given hypothesis set, does it exists a learning machine capable to learn it? The answer to such a question all ws to distinguish between learnable and non learnable problem. The remaining question being does it xists some learning machine capable of learning any learnable set. We know since [12] that learning is closely related to the app roximation theory, generalized spline theory, regularization and behind the notion of reproducing kernel Hilbert space ( r:k:h:s). This framework is based on the minimization of the empirical cost and a stabilizer ( i.e. a norm is some Hilbert space). Then, under these conditions, the solution of the learning task is a line ar combination of some positive kernel whose shape depends on the nature of the stabilizer. This solution is characterized by strong and nice properties such as universal consistency. But within this framework it remains some gap between theory and some practical solution implemented by practitioners. For instance in r:k:h:s kernels are positive. Some practitioner uses hyperbolic ta ngent kernel tanh(w>x + w0) and it is not a positive kernel: but it works. An other example is given by practitioner using non hilbertian framework. The sparcity upholder uses ab olute values such as R jf jd or Pj j j j: these areL1 norms. They are not hilbertian. Other escape the hilbertian approximation orthodoxy by introducing prior knowledge ( i.e. a stabilizer) through information type criteria that are no t rms. This paper aims at revealing some underlying hypothesis of t he learning task stretching the reproducing kernel Hilbert space framework. To do so we begin with review ing some learning principle. We will stress that the hilbertian nature of the hypothesis set is not neces sary while the reproducing property is. This leads us to define a non hilbertian framework for reproducing kerne l allowing non positive kernel, non hilbertian norms and other kinds of stabilizers.