When the data is noiseless, regularization is harmful. We emphasize the necessity of prior information for effective regularization. By counting functions on a discretized grid, we develop a framework for incorporating prior knowledge about the target function into the learning process. Using this framework, we derive a direct connection between smoothness priors and Tikhonov regularization, in addition to the regularization terms implied by other priors. We then consider using maximum likelihood to predict non-stationary noise variance in time series.
Maximum likelihood leads to systematic errors that favor lower variance.
We discuss the systematic correction of these errors. We develop stochastic and deterministic techniques for density estimation based on approximating the distribution function, thus placing density estimation within the supervised learning framework. In general, we use multilayer neural networks to illustrate our methods.