Above are papers related to the following paper: [Paper_ID] Paper #14807. 	[Title] Exponentially many local minima for single neurons [Abstract] We show that for a single neuron with the logistic function as the transfer function the number of local minima of the error function based on the square loss can grow exponentially in the dimension. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #289088. 	[Title] Critical Points for Least-Squares Problems Involving Certain Analytic Functions, with Applications to Sigmoidal Nets [Abstract] This paper deals with nonlinear least-squares problems involving the fitting to data of parameterized analytic functions. For generic regression data, a general result establishes the countability, and under stronger assumptions finiteness, of the set of functions giving rise to critical points of the quadratic loss function. In the special case of what are usually called "single-hidden layer neural networks," which are built upon the standard sigmoidal activation tanh(x) (or equivalently (1 + e x ) 1 ), a rough upper bound for this cardinality is provided as well. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #123556. 	[Title] BACKPROPAGATION SEPARATES WHERE PERCEPTRONS DO [Abstract] Feedforward nets with sigmoidal activation functions are often designed by minimizing a cost criterion. It has been pointed out before that this technique may be outperformed by the classical perceptron learning rule, at least on some problems. In this paper, we show that no such pathologies can arise if the error criterion is of a threshold LMS type, i.e., is zero for values "beyond" the desired target values. More precisely, we show that if the data are linearly separable, and one considers nets with no hidden neurons, then an error function as above cannot have any local minima that are not global. Simulations of networks with hidden units are consistent with these results, in that often data which can be classified when minimizing a threshold LMS criterion may fail to be classified when using instead a simple LMS cost. In addition, the proof gives the following stronger result, under the stated hypotheses: the continuous gradient adjustment procedure is such that from any initial weight configuration a separating set of weights is obtained in finite time. This is a precise analogue of the Perceptron Learning Theorem. The results are then compared with the more classical pattern recognition problem of threshold LMS with linear activations, where no spurious local minima exist even for nonseparable data: here it is shown that even if using the threshold criterion, such bad local minima may occur, if the data are not separable and sigmoids are used. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1111240. 	[Title] On the Distribution of Performance from Multiple Neural Network Trials, On the Distribution of Performance [Abstract] Andrew D. Back was with the Department of Electrical and Computer Engineering, University of Queensland. St. Lucia, Australia. He is now with the Brain Information Processing Group, Frontier Research Program, RIKEN, The Institute of Physical and Chemical Research, 2-1 Hirosawa, Wako-shi, Saitama 351-01, Japan Abstract The performance of neural network simulations is often reported in terms of the mean and standard deviation of a number of simulations performed with different starting conditions. However, in many cases, the distribution of the individual results does not approximate a Gaussian distribution, may not be symmetric, and may be multimodal. We present the distribution of results for practical problems and show that assuming Gaussian distributions can significantly affect the interpretation of results, especially those of comparison studies. For a controlled task which we consider, we find that the distribution of performance is skewed towards better performance for smoother target functions and skewed towards worse performance Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #141160. 	[Title] LINEAR SYSTEMS WITH SIGN-OBSERVATIONS [Abstract] This paper deals with systems that are obtained from linear time-invariant continuous-or discrete-time devices followed by a function that just provides the sign of each output. Such systems appear naturally in the study of quantized observations as well as in signal processing and neural network theory. Results are given on observability, minimal realizations, and other system-theoretic concepts. Certain major differences exist with the linear case, and other results generalize in a surprisingly straightforward manner. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #193931. 	[Title] What Size Neural Network Gives Optimal Generalization? Convergence Properties of Backpropagation [Abstract] Technical Report UMIACS-TR-96-22 and CS-TR-3617 Institute for Advanced Computer Studies University of Maryland College Park, MD 20742 Abstract One of the most important aspects of any machine learning paradigm is how it scales according to problem size and complexity. Using a task with known optimal training error, and a pre-specified maximum number of training updates, we investigate the convergence of the backpropagation algorithm with respect to a) the complexity of the required function approximation, b) the size of the network in relation to the size required for an optimal solution, and c) the degree of noise in the training data. In general, for a) the solution found is worse when the function to be approximated is more complex, for b) oversized networks can result in lower training and generalization error in certain cases, and for c) the use of committee or ensemble techniques can be more beneficial as the level of noise in the training data is increased. For the experiments we performed, we do not obtain the optimal solution in any case. We further support the observation that larger networks can produce better training and generalization error using a face recognition example where a network with many more parameters than training points generalizes better than smaller networks. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #193932. 	[Title] Lessons in Neural Network Training: Overfitting Lessons in Neural Network Training: Overfitting May be Harder [Abstract] For many reasons, neural networks have become very popular AI machine learning models. Two of the most important aspects of machine learning models are how well the model generalizes to unseen data, and how well the model scales with problem complexity. Using a controlled task with known optimal training error, we investigate the convergence of the backpropagation (BP) algorithm. We find that the optimal solution is typically not found. Furthermore, we observe that networks larger than might be expected can result in lower training and generalization error. This result is supported by another real world example. We further investigate the training behavior by analyzing the weights in trained networks (excess degrees of freedom are seen to do little harm and to aid convergence), and contrasting the interpolation characteristics of multi-layer perceptron neural networks (MLPs) and polynomial models (overfitting behavior is very different the MLP is often biased towards smoother solutions). Finally, we analyze relevant theory outlining the reasons for significant practical differences. These results bring into question common beliefs about neural network training regarding convergence and optimal network size, suggest alternate guidelines for practical use (lower fear of excess degrees of freedom), and help to direct future work (e.g. methods for creation of more parsimonious solutions, importance of the MLP/BP bias and possibly worse performance of improved training algorithms). Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #50354. 	[Title] BACKPROPAGATION CAN GIVE RISE TO SPURIOUS LOCAL MINIMA EVEN FOR NETWORKS WITHOUT HIDDEN LAYERS [Abstract] We give an example of a neural net without hidden layers and with a sigmoid transfer function, together with a training set of binary vectors, for which the sum of the squared errors, regarded as a function of the weights, has a local minimum which is not a global minimum. The example consists of a set of 125 training instances, with four weights and a threshold to be learnt. We do not know if substantially smaller binary examples exist. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #6775. 	[Title] Statistical Evaluation of Neural Network Experiments: Minimum Requirements and Current Practice [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #264347. 	[Title] Relative Loss Bounds for Multidimensional Regression Problems [Abstract] We study on-line generalized linear regression with multidimensional outputs, i.e., neural networks with multiple output nodes but no hidden nodes. We allow at the final layer transfer functions such as the softmax function that need to consider the linear activations to all the output neurons. We also use a parameterization function which transforms parameter vectors maintained by the algorithm into the actual weights. The on-line algorithm we consider updates the parameters in an additive manner, analogous to the delta rule, but because the actual weights are obtained via the possibly nonlinear parameterization function they may behave in a very different manner. Our approach is based on applying the notion of a matching loss function in two different contexts. First, we measure the loss of the algorithm in terms of the loss that matches the transfer function used to produce the outputs. Second, the loss function that matches the parameterization function can be used both as a measure of distance between models in motivating the update rule of the algorithm and as a potential function in analyzing its relative performance compared to an arbitrary fixed model. As a result, we have a unified treatment that generalizes earlier results for the gradient descent and exponentiated gradient algorithms to multidimensional outputs, including multiclass logistic regression. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #193918. 	[Title] A Unifying View of Some Training Algorithms for Multilayer Perceptrons with FIR Filter Synapses [Abstract] Recent interest has come about in deriving various neural network architectures for modelling time-dependent signals. A number of algorithms have been published for multilayer perceptrons with synapses described by finite impulse response (FIR) and infinite impulse response (IIR) filters (the latter case is also known as Locally Recurrent Globally Feedforward Networks). The derivations of these algorithms have used different approaches in calculating the gradients, and in this note, we present a short, but unifying account of how these different algorithms compare for the FIR case, both in derivation, and performance. New algorithms are subsequently presented. Simulation results have been performed to benchmark these algorithms. In this note, results are compared for the Mackey-Glass chaotic time series against a number of other methods including a standard multilayer perceptron, and a local approximation method. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1153183. 	[Title]  [Abstract] General convergence results for linear discriminant updates Abstract The problem of learning linear discriminant concepts can be solved by various mistake-driven update procedures, including the Winnow family of algorithms and the well-known Perceptron algorithm. In this paper we define the general class of quasi-additive algorithms, which includes Perceptron and Winnow as special cases. We give a single proof of convergence that covers much of this class, including both Perceptron and Winnow but also many novel algorithms. Our proof introduces a generic measure of progress that seems to capture much of when and how these algorithms converge. Using this measure, we develop a simple general technique for proving mistake bounds, which we apply to the new algorithms as well as existing algorithms. When applied to known algorithms, our technique automatically produces close variants of existing proofs (and we generally obtain the known bounds, to within constants) thus showing, in a certain sense, that these seem ingly diverse results are fundamentally isomorphic. Please summarise all the given information.

Paper #14807 has the title `Exponentially many local minima for single neurons  
` and the abstract `We show that for a single neuron with the logistic function as the transfer function the number of local minima of the error function based on the square loss can grow exponentially in the dimension.
`.
Question: Which category should Paper #14807 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].


Paper #14807 has the title `Exponentially many local minima for single neurons  
` and the abstract `We show that for a single neuron with the logistic function as the transfer function the number of local minima of the error function based on the square loss can grow exponentially in the dimension.
`.
Question: Which category should Paper #14807 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].

This paper should be classified into: Theory