nils-herrmann/clean_scirepeval huggingface dataset:
Missing Value Estimation Using Mixture of PCAs

We apply mixture of principal component analyzers (MPCA) to missing value estimation problems. A variational Bayes (VB) method for MPCA with missing values is developed. The missing values are regarded as hidden variables aud their estimation is done simultaneously with the parameter estimation. It is found that VB method is better than maximum likelihood method by using artificial data. We also applied our method to DNA microarray data and the performance outweighed the conventional k-nearest neighbor method.
Infinite dimensional exponential families by reproducing kernel Hilbert spaces

The purpose of this paper is to propose a method of constructing exponential families of Hilbert manifold, on which estimation theory can be built. Although there have been works on infinite dimensional exponential families of Banach manifolds (Pistone and Sempi, 1995; Gibilisco and Pistone, 1998; Pistone and Rogantin, 1999), they are not appropriate to discuss statistical estimation with finite number of samples; the likelihood function with finite samples is not continuous on the manifold. In this paper we use a reproducing kernel Hilbert space as a functional space for constructing an exponential manifold. A reproducing kernel Hilbert space is defined as a Hilbert space of functions such that evaluation of a function at an arbitrary point is a continuous functional on the Hilbert space. Since we can discuss the value of a function with this space, it is very natural to use a manifold associated with a reproducing kernel Hilbert space as a basis of estimation theory. We focus on the maximum likelihood estimation (MLE) with the exponential manifold of a reproducing kernel Hilbert space. As in many non-parametric estimation methods, straightforward extension of MLE to an infinite dimensional exponential manifold suffers the problem of ill-posedness caused by the fact that the estimator should be chosen from the infinite dimensional space with only finite number of constraints given by the data. To solve this problem, a pseudo-maximum likelihood method is proposed by restricting the infinite dimensional manifold to a series of finite dimensional submanifolds, which enlarge as the number of samples increases. Some asymptotic results in the limit of infinite samples are shown, including the consistency of the pseudo-MLE.
Multiresolution Kernels

We present in this work a new methodology to design kernels on data which is structured with smaller components, such as text, images or sequences. This methodology is a template procedure which can be applied on most kernels on measures and takes advantage of a more detailed “bag of components” representation of the objects. To obtain such a detailed description, we consider possible decompositions of the original bag into a collection of nested bags, following a prior knowledge on the objects’ structure. We then consider these smaller bags to compare two objects both in a detailed perspective, stressing local matches between the smaller bags, and in a global or coarse perspective, by considering the entire bag. This multiresolution approach is likely to be best suited for tasks where the coarse approach is not precise enough, and where a more subtle mixture of both local and global similarities is necessary to compare objects. The approach presented here would not be computationally tractable without a factorization trick that we introduce before presenting promising results on an image retrieval task.
Over-fitting behavior of Gaussian unit under Gaussian noise

In the training of neural networks and radial basis function networks under noisy environment, it is important to know how the network over-fits to the noise in the given data since it is directly related to the model selection and regularization problem. In this paper, we firstly derive a probabilistic upper bound for the degree of over-fitting. By applying this result, we consider the over-fitting behavior of a Gaussian unit, which is trained under Gaussian noise, and we show that the probability that the width parameter of the Gaussian unit takes an extremely small value in training under Gaussian noise goes to one as the number of samples goes to infinity.
A kernel method for hierarchical and non-hierarchical clustering

In cluster analysis, choosing a distance measure among given data is not straightforward. For defining a similarity matrix in hierarchical clustering and some types of non-hierarchical clustering such as spectral clustering (), several distance measures such as Pearson correlation, Euclidean distance, and nonparametric measures have been widely used. The result of clustering, however, often depends on a specific choice of the measure. If our purpose is to extract a group of variables, for which we cannot presume the type of dependence, a specific choice of measure may not capture the dependence of variables. This paper proposes a novel approach to define a similarity matrix, which can incorporate any nonlinear correlation of variables. Our method uses kernel Hilbert spaces, which contain a sufficiently rich class of nonlinear functions of variables. We define the similarity of two variables based on the theory of kernel Hilbert spaces and covariance operators (). This similarity represents the sum of all the nonlinear correlations in principle, and provides a more reasonable way for defining the similarity or dependence of two random variables than existing measures, which extract only partial information of variables.
Likelihood ratio of unidentifiable models and multilayer neural networks

This paper discusses the behavior of the maximum likelihood estimator (MLE), in the case that the true parameter cannot be identified uniquely. Among many statistical models with unidentifiability, neural network models are the main concern of this paper. It is known in some models with unidentifiability that the asymptotics of the likelihood ratio of the MLE has an unusually larger order. Using the framework of locally conic models put forth by Dacunha-Castelle and Gassiat as a generalization of Hartigan's idea, a useful sufficient condition of such larger orders is derived. This result is applied to neural network models, and a larger order is proved if the true function is given by a smaller model. Also, under the condition that the model has at least two redundant hidden units, a log n lower bound for the likelihood ratio is derived.
Statistical Estimation in Singular Models.

パラメトリックな統計モデルや学習機械を関数空間の中で考えたときに, 滑らかでない点を持つようなモデルをここでは特異モデルと呼ぶ. 多層パーセプトロン, ミクスチャモデルやARMA, HMMなどがその例となっている. このようなモデルでは, 真のパラメータがその特異点にあるとき, 推定や学習にさまざまな複雑で興味深い現象がみられる. 本論文では, 特異モデルに生じる統計的現象とその理論的解析について, 接錐による見方を中心に述べる.
A general upper bound of likelihood ratio for regression

This paper discusses the likelihood ratio test statistics (LRTS) in regression problems, and derives a general upper bound of the asymptotic order of LRTS for the sample size n. In some cases of estimation, where the true parameter is not identifiable, the LRTS diverges to infinity asymptotically. It is also known that the LRTS of some nonlinear models has a lower bound of the order logn. This paper shows logn gives an upper bound of the asymptotic order of regression under very general assumptions, which are satisfied by many practical probability models including the Gaussian noise and the binary regression.
No . 858 November 07 , 2002 A general upper bound of likelihood ratio in binary regression

We derive a general upper bound of the asymptotic order of the likelihood ratio test statistics (LRTS) for binary regression. It has been known that in some unidentifiable cases of nonlinear regression the LRTS may diverge to infinity asymptotically and have a log n lower bound for the sample size n. This paper shows an upper bound of log n under the very general assumptions of the finiteness of VC dimension of the regressor class and the boundedness of the assumed true regressor.
Asymptotic Theory of Locally Conic Models and its Application to Multilayer Neural Networks

This paper discusses the maximum likelihood estimation in a statistical model with unidentiÞability, using the framework of conic singularity. The likelihood ratio may diverge in unidentiÞable cases, though in regular cases it converges to a χ distribution. A useful sufficient condition of such divergence is obtained, and is applied to neural networks. The exact order for multilayer perceptrons is also discussed.
Likelihood Ratio of Unidenti Þ able Models and Multilayer Neural Networks

This paper discusses the behavior of the maximum likelihood estimator, when the true parameter cannot be identiÞed uniquely. Among many statistical models with unidentiÞability, neural network models are the main concern of this paper. The set of unidentiÞable true parameters is formulated as a conic singularity of the model, which is embedded in an inÞnite dimensional space of probability density functions. It has been known in some models with unidentiÞability the asymptotics of the likelihood ratio of MLE has an unusually larger order. Following Hartigans idea, the likelihood ratio of MLE is described by the supremum of an empirical process over a set of functions, and a useful sufficient condition of such larger orders is derived. This result is applied to neural network models, and a larger order is observed if the true function is realized by a network with a smaller number of hidden units than the model. A stronger lower bound of the order of likelihood ratio is also derived on condition that there are at least two redundant hidden units to realize the true function.
Asymptotic Theory of Locally Conic Models and its Applications to Neural Networks

Multilayer neural networks have a problem of unidentiÞability in its parameterization. If a network has surplus hidden units to realize a target function, the parameters to give the function consist of a high dimensional subset. Many of usual statistical views fail in such cases. This paper discusses the likelihood ratio of the maximum likelihood estimation in unidentiÞable cases, using the framework of locally conic models. We derive a sufficient condition that the likelihood ratio has a larger order than usual Op(1). The exact order of the likelihood ratio of multilayer perceptrons is derived, and a new regularization scheme is proposed to overcome the strong overÞtting.
Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons

The natural gradient learning method is known to have ideal performances for on-line training of multilayer perceptrons. It avoids plateaus, which give rise to slow convergence of the backpropagation method. It is Fisher efficient, whereas the conventional method is not. However, for implementing the method, it is necessary to calculate the Fisher information matrix and its inverse, which is practically very difficult. This article proposes an adaptive method of directly obtaining the inverse of the Fisher information matrix. It generalizes the adaptive Gauss-Newton algorithms and provides a solid theoretical justification of them. Simulations show that the proposed adaptive method works very well for realizing natural gradient learning.
Inference and Learning with Graphical Models

Intelligent systems which work in the real world cooperating with humans should be able to treat complex, uncertain, dynamic, multi-modal information in the real world environment. Probability distributions with graphical structure (Graphical Models) are recently considered as a promising fundamental tools to represent and process such information. From this point of view, toward a theoretical and algorithmic foundation for real world intelligence, we are investigating inference and learning algorithms for several kinds of graphical models such as Bayesian networks, probabilistic constraint program, mixture models, multivariate models. Novel learning frameworks suited for real world intelligence are also investigated. Here we introduce some remarkable results from our recent researches.
Statistical Analysis of UnidentiÞable Models and its Application to Multilayer Neural Networks

This paper discusses the maximum likelihood estimator of the parametric model that has lack of identiÞability in low dimensional subsets in the parameter space. Among many statistical models with unidentiÞability, neural network models are the main concern of this paper. The unidentiÞable true parameter is formulated as a conic singularity of the model embedded in an inÞnite dimensional space of probability density functions. Following Hartigan’s idea, the likelihood ratio of the maximum likelihood estimator is described by the supremum of an empirical process over a set of functions. It has been known in some models the asymptotics of the likelihood ratio has an unusually larger order. A useful sufficient condition of the larger order is shown, and applied to neural networks. The order of the asymptotic likelihood ratio are shown in various cases of multilayer perceptrons.
Dynamics of Batch Learning in Multilayer Networks

This paper investigates the dynamics of batch learning of multilayer neural networks in the asymptotic case where the number of training data is much larger than the number of parameters. We consider regression problems assuming noisy output data. First, we present experimental results on the behavior in the steepest descent learning of multilayer perceptrons and three-layer linear neural networks. We see in these results that strong overtraining, which is the increase of the generalization error in training, occurs if the model has surplus hidden units to realize the target function. Next, to analyze overtraining from the theoretical viewpoint, we consider the solution of the steepest descent learning equation of a three-layer linear neural network, and prove that a network with surplus hidden units shows overtraining. From this theoretical analysis, we can see that overtraining is not a feature observed in the nal stage of learning, but in an intermediate time interval.
An Efficient Learning Algorithm Using Naturla Gradient and Second Order Information of Error Surface

Natural gradient learning algorithm, which originated from information geometry, is known to provide a good solution for the problem of slow learning speed of gradient descent learning methods. Whereas the natural gradient learning algorithm is inspired from the geometric structure of the space of learning systems, there have been other approaches to acceleration of learning by using the second order information of error surface. Although the second order methods cannot give as successful solutions as the natural gradient learning method, their results showed the usefulness of the second order information of error surface in the learning process. In this paper, we develop a method of combining these two different approaches to propose a more efficient learning algorithm. At each learning step, we calculate a search direction by means of the natural gradient. When we apply the search direction to parameter-updating process, the second order information of error surface is applied to determine an efficient learning rate. Through a simple experiment on a real world problem, we confirmed that the proposed learning algorithm show faster convergence than the pure natural gradient learning algorithm.
Active Learning in Multilayer

We propose an active learning method with hidden-unit reduction, which is devised specially for multilayer perceptrons (MLP). First, we review our active learning method, and point out that many Fisher-information-based methods applied to MLP have a critical problem: the information matrix may be singular. To solve this problem, we derive the singularity condition of an information matrix, and propose an active learning technique that is applicable to MLP. Its e ectiveness is veri ed through experiments.
Statistical active learning in multilayer perceptrons

This paper proposes new methods for generating input locations actively in gathering training data, aiming at solving problems unique to multilayer perceptrons. One of the problems is that optimum input locations, which are calculated deterministically, sometimes distribute densely around the same point and cause local minima in backpropagation training. Two probabilistic active learning methods, which utilize the statistical variance of locations, are proposed to solve this problem. One is parametric active learning and the other is multipoint-search active learning. Another serious problem in applying active learning to multilayer perceptrons is that a Fisher information matrix can be singular, while many methods, including the proposed ones, assume its regularity. A technique of pruning redundant hidden units is proposed to keep the Fisher information matrix regular. Combined with this technique, active learning can be applied stably to multilayer perceptrons. The effectiveness of the proposed methods is demonstrated through computer simulations on simple artificial problems and a real-world problem of color conversion.
Learning in neural networks and an integrable system (Applied Mathematics of Discrete Integrable Systems)

This paper investigates the dynamics of batch learning of multilayer neural networks in the asymptotic case where the number of training data is much larger than the number of parameters. First, we present experimental results on the behavior in the steepest descent learning of multilayer perceptrons and three-layer linear neural networks. We see in these results that strong overtraining, which is the increase of generalization error in training, occurs if the model has surplus hidden units to realize the target function. Next, to analyze overtraining from the theoretical viewpoint, we analyze the steepest descent learning equation of a three-layer linear neural network, and theoretically show that a network with surplus hidden units presents overtraining. From this theoretical analysis, we can see that overtraining is not a feature observed in the nal stage of learning, but it occurs in an intermediate time interval.
Generalization Error of Limear Neural Networks in Unidentifiable Cases

The statistical asymptotic theory is often used in theoretical results in computational and statistical learning theory. It describes the limiting distribution of the maximum likelihood estimator (MLE) as an normal distribution. However, in layered models such as neural networks, the regularity condition of the asymptotic theory is not necessarily satisfied. The true parameter is not identifiable, if the target function can be realized by a network of smaller size than the size of the model. There has been little known on the behavior of the MLE in these cases of neural networks. In this paper, we analyze the expectation of the generalization error of three-layer linear neural networks, and elucidate a strange behavior in unidentifiable cases. We show that the expectation of the generalization error in the unidentifiable cases is larger than what is given by the usual asymptotic theory, and dependent on the rank of the target function.
Generalization Error of Linear Neural Networks in Unidentiable Cases

The statistical asymptotic theory is often used in theoreti- cal results in computational and statistical learning theory. It describes the limiting distribution of the maximum likelihood estimator (MLE) as an normal distribution. However, in layered models such as neural networks, the regularity condition of the asymptotic theory is not nec- essarily satised. The true parameter is not identiable, if the target function can be realized by a network of smaller size than the size of the model. There has been little known on the behavior of the MLE in these cases of neural networks. In this paper, we analyze the expectation of the generalization error of three-layer linear neural networks, and elucidate a strange behavior in unidentiable cases. We show that the expectation of the generalization error in the unidentiable cases is larger than what is given by the usual asymptotic theory, and dependent on the rank of the target function.
Local minima and plateaus in multilayer neural networks

Local minima and plateaus pose a serious problem in learning of neural networks. We investigate the geometric structure of the parameter space of three-layer perceptrons in order to show the existence of local minima and plateaus. It is proved that a critical point of the model with H-1 hidden units always gives a critical point of the model with H hidden units. Based on this result, we prove that the critical point corresponding to the global minimum of a smaller model can be a local minimum or a saddle point of the larger model. We give a necessary and sufficient condition for this. The results are universal in the sense that they do not use special properties of target, loss functions, and activation functions, but only use the hierarchical structure of the model.
Effect of Batch Learning in Multilayer Neural Networks

This paper discusses batch gradient descent learning in mul-tilayer networks with a large number of statistical training data. We emphasize on the diierence between regular cases, where the prepared model has the same size as the true function , and overrealizable cases, where the model has surplus hidden units to realize the true function. First, experimental study on multilayer perceptrons and linear neural networks (LNN) shows that batch learning induces strong overtrain-ing on both models in overrealizable cases, which means the degrade of generalization error by surplus units can be alleviated. We theoretically analyze the dynamics in LNN, and show that this overtraining is caused by shrinkage of the parameters corresponding to surplus units.
Dynamics of Batch Learning in Multilayer Neural Networks

We discuss the dynamics of batch learning of multilayer neural networks in the asymptotic limit, where the number of trining data is much larger than the number of parameters, emphasizing on the parameterization redundancy in overrealizable cases. In addition to showing experimental results on overtraining in multilayer perceptrons and three-layer linear neural networks, we theoretically prove the existence of overtraining in overrealizable cases of the latter model.
Active Learning in Multilayer Perceptrons

We propose an active learning method with hidden-unit reduction. which is devised specially for multilayer perceptrons (MLP). First, we review our active learning method, and point out that many Fisher-information-based methods applied to MLP have a critical problem: the information matrix may be singular. To solve this problem, we derive the singularity condition of an information matrix, and propose an active learning technique that is applicable to MLP. Its effectiveness is verified through experiments.
Probabilistic design of layered neural networks based on their unified framework

Proposes three ways of designing artificial neural networks based on a unified framework and uses them to develop new models. First, the authors show that artificial neural networks can be understood as probability density functions with parameters. Second, the authors propose three design methods for new models: a method for estimating the occurrence probability of the inputs, a method for estimating the variance of the outputs, and a method for estimating the simultaneous probability of inputs and outputs. Third, the authors design three new models using the proposed methods: a neural network with occurrence probability estimation, a neural network with output variance estimation, and a probability competition neural network. The authors' experimental results show that the proposed neural networks have important abilities in information processing; they can tell how often a given input occurs, how widely the outputs are distributed, and from what kinds of inputs a given output is inferred.