nils-herrmann/clean_scirepeval huggingface dataset:
Combining Conjugate Direction Methods with Stochastic Approximation of Gradients

The method of conjugate directions provides a very eective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. Here we explore ideas from conjugate gradient in the stochastic (online) setting, using fast Hessian-gradient products to set up low-dimensional Krylov subspaces within individual mini-batches. In our benchmark experiments the resulting online learning algorithms converge orders of magnitude faster than ordinary stochastic gradient descent.
Solving Noisy Linear Operator Equations by Gaussian Processes: Application to Ordinary and Partial Differential Equations

We formulate the problem of solving stochastic linear operator equations in a Bayesian Ganssian process (GP) framework. The solution is obtained in the spirit of a collocation method based on noisy evaluations of the target function at randomly drawn or deliberately chosen points. Prior knowledge about the solution is encoded by the covariance kernel of the GP. As in GP regression, analytical expressions for the mean and variance of the estimated target function are obtained, from which the solution of the operator equation follows by a manipulation of the kernel. Linear initial and boundary value constraints can be enforced by embedding the non-parametric model in a form that automatically satisfies the boundary conditions. The method is illustrated on a noisy linear first-order ordinary differential equation with initial condition and on a noisy second-order partial differential equation with Dirichlet boundary conditions.
Correspondence Analysis for Visualizing Interplay of Pitch Class, Key, and Composer

We apply correspondence analysis for visualization of interdependence of pitch class & key and key & composer. A co-occurrence matrix of key & pitch class frequencies is extracted from score (Bach’s WTC). Keys are represented as high-dimensional pitch class vectors. Correspondence analysis then projects keys on a planar ikeyscapei. Vice versa, on ipitchscapesi pitch classes can also be embedded in the key space. In both scenarios a homogenous circle of fths emerges in the scapes. We employ biplots to embed keys and pitch classes in the keyscape to visualize their interdependence. After a change of co-ordinates the four-dimensional biplots can be interpreted as a conguration on a torus, closely resembling results from music theory and experiments in listener models. In conjunction with spectral analysis, correspondence analysis constitutes a cognitive auditory model. Correspondence analysis of the co-occurrence table of intensities of keys and pitch classes lets the circle of fths evolve in the pitchscape. This model works on digitized recorded music, does not require averaging or normalization of the data, and does not implicitly use circularity inherent in the model. Statistics on key preference in composers yields a composer & key cooccurrence matrix. Then istylescapesi visualize relations between musical styles of particular composers and schools. The Biplotting technique links stylistic characteristics to favored keys. Interdependence of composers and schools is meaningfully visualized according to their key preferences.
Kernel Matrix Completion by Semidefinite Programming

We consider the problem of missing data in kernel-based learning algorithms. We explain how semidefinite programming can be used to perform an approximate weighted completion of the kernel matrix that ensures positive semidefiniteness and hence Mercer's condition. In numerical experiments we apply a support vector machine to the XOR classification task based on randomly sparsified kernel matrices from a polynomial kernel of degree 2. The approximate completion algorithm leads to better generalisation and to fewer support vectors as compared to a simple spectral truncation method at the cost of considerably longer runtime.We argue that semidefinite programming provides an interesting convex optimisation framework for machine learning in general and for kernel-machines in particular.
Towards stochastic conjugate gradient methods

The method of conjugate gradients provides a very effective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. We explore a number of ways to adopt ideas from conjugate gradient in the stochastic setting, using fast Hessian-vector products to obtain curvature information cheaply. In our benchmark experiments the resulting highly scalable algorithms converge about an order of magnitude faster than ordinary stochastic gradient descent.
Conjugate Directions for Stochastic Gradient Descent

The method of conjugate gradients provides a very effective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. Here we explore ideas from conjugate gradient in the stochastic (online) setting, using fast Hessian-gradient products to set up low-dimensional Krylov subspaces within individual mini-batches. In our benchmark experiments the resultingonline learningalg orithms converge orders of magnitude faster than ordinary stochastic gradient descent.
Stable Adaptive Momentum for Rapid Online Learning in Nonlinear Systems

We consider the problem of developing rapid, stable, and scalable stochastic gradient descent algorithms for optimisation of very large nonlinear systems. Based on earlier work by Orr et al. on adaptive momentum--an efficient yet extremely unstable stochastic gradient descent algorithm--we develop a stabilised adaptive momentum algorithm that is suitable for noisy nonlinear optimisation problems. The stability is improved by introducing a forgetting factor 0 ? ? ? 1 that smoothes the trajectory and enables adaptation in non-stationary environments. The scalability of the new algorithm follows from the fact that at each iteration the multiplication by the curvature matrix can be achieved in O (n) steps using automatic differentiation tools. We illustrate the behaviour of the new algorithm on two examples: a linear neuron with squared loss and highly correlated inputs, and a multilayer perceptron applied to the four regions benchmark task.
Gaussian process regression: active data selection and test point rejection

We consider active data selection and test point rejection strategies for Gaussian process regression based on the variance of the posterior over target values. Gaussian process regression is viewed as transductive regression that provides target distributions for given points rather than selecting an explicit regression function. Since not only the posterior mean but also the posterior variance are easily calculated we use this additional information to two ends: active data selection is performed by either querying at points of high estimated posterior variance or at points that minimize the estimated posterior variance averaged over the input distribution of interest or (in a transductive manner) averaged over the test set. Test point rejection is performed using the estimated posterior variance as a confidence measure. We find that, for both a two-dimensional toy problem and a real-world benchmark problem, the variance is a reasonable criterion for both active data selection and test point rejection.
Gaussian Process Regression: Active Data Selection and Test Point Rejection

We consider active data selection and test point rejection strategies for Gaussian process regression based on the variance of the posterior over target values. Gaussian process regression is viewed as transductive regression that provides target distributions for given points rather than selecting an explicit regression function. Since not only the posterior mean but also the posterior variance are easily calculated we use this additional information to two ends: active data selection is performed by either querying at points of high estimated posterior variance or at points that minimize the estimated posterior variance averaged over the input distribution of interest or (in a transductive manner) averaged over the test set. Test point rejection is performed using the estimated posterior variance as a confidence measure. We find that, for both a two-dimensional toy problem and a real-world benchmark problem, the variance is a reasonable criterion for both active data selection and test point rejection.
Gaussian Process Regression: Active Data Selection and Test Point

We consider active data selection and test point rejection strategies for Gaussian process regression based on the variance of the posterior over target values. Gaussian process regression is viewed as transductive regression that provides target distributions for given points rather than selecting an explicit regression function. Since not only the posterior mean but also the posterior variance are easily calculated we use this additional information to two ends: Active data selection is performed by either querying at points of high estimated posterior variance or at points that minimize the estimated posterior variance averaged over the input distribution of interest or - in a transductive manner - averaged over the test set. Test point rejection is performed using the estimated posterior variance as a confidence measure. We find for both a two-dimensional toy problem and for a real-world benchmark problem that the variance is a reasonable criterion for both active data selection and test point rejection.
A Stochastic Self-Organizing Map for Proximity Data

We derive an efficient algorithm for topographic mapping of proximity data (TMP), which can be seen as an extension of Kohonen's self-organizing map to arbitrary distance measures. The TMP cost function is derived in a Baysian framework of folded Markov chains for the description of autoencoders. It incorporates the data by a dissimilarity matrix and the topographic neighborhood by a matrix of transition probabilities. From the principle of maximum entropy, a nonfactorizing Gibbs distribution is obtained, which is approximated in a mean-field fashion. This allows for maximum likelihood estimation using an expectation-maximization algorithm. In analogy to the transition from topographic vector quantization to the self-organizing map, we suggest an approximation to TMP that is computationally more efficient. In order to prevent convergence to local minima, an annealing scheme in the temperature parameter is introduced, for which the critical temperature of the first phase transition is calculated in terms of and . Numerical results demonstrate the working of the algorithm and confirm the analytical results. Finally, the algorithm is used to generate a connection map of areas of the cat's cerebral cortex.
Bayesian Learning in Reproducing KernelHilbert Spaces { The Usefulness of theBayes

From a Bayesian perspective Support Vector Machines can be viewed as constructing a hypothesis using the center of the largest possible hypersphere which can be placed inside the version space, i.e. the space of all consistent hypotheses given a training set. The points of contact between this hypersphere and the boundaries of the version space de ne the support vectors. An alternative and potentially better approach is to construct the hypothesis using the whole version space. This can be achieved by using the Bayes point which is the midpoint of the region of intersection of all hyperplanes bisecting the version space into two volumes of equal size. It is known that the center of mass of the version space approximates the Bayes point [19]. Our approach to estimating the center of mass is to follow the trajectory of a billiard in the version space. We present experimental results which indicate that our algorithm consistently outperforms Support Vector Machines.
An Annealed Self-Organizing Map for Source Channel Coding

We derive and analyse robust optimization schemes for noisy vector quantization on the basis of deterministic annealing. Starting from a cost function for central clustering that incorporates distortions from channel noise we develop a soft topographic vector quantization algorithm (STVQ) which is based on the maximum entropy principle and which performs a maximum-likelihood estimate in an expectation-maximization (EM) fashion. Annealing in the temperature parameter β leads to phase transitions in the existing code vector representation during the cooling process for which we calculate critical temperatures and modes as a function of eigenvectors and eigenvalues of the covariance matrix of the data and the transition matrix of the channel noise. A whole family of vector quantization algorithms is derived from STVQ, among them a deterministic annealing scheme for Kohonen's self-organizing map (SOM). This algorithm, which we call SSOM, is then applied to vector quantization of image data to be sent via a noisy binary symmetric channel. The algorithm's performance is compared to those of LBG and STVQ. While it is naturally superior to LBG, which does not take into account channel noise, its results compare very well to those of STVQ, which is computationally much more demanding.
Phase Transitions in Soft Topographic Vector Quantization

We have developed an algorithm (STVQ) for the optimization of neighborhood preserving maps by applying deterministic annealing to an energy function for topographic vector quantization. The combinatorial optimization problem is solved by introducing temperature dependent fuzzy assignments of data points to cluster centers and applying an EM-type algorithm at each temperature while annealing. The annealing process exhibits phase transitions in the cluster representation for which we calculate critical modes and temperatures expressed in terms of the neighborhood function and the covariance matrix of the data. In particular, phase transitions corresponding to the automatic selection of feature dimensions are explored analytically and numerically for finite temperatures. Results are related to those obtained earlier for Kohonen's SOM-algorithm which can be derived as an approximation to STVQ. The deterministic annealing approach makes it possible to use the neighborhood function solely to encode desired neighborhood relations. The working of the annealing process is visualized by showing the effects of “heating” on the topological structure of a two-dimensional map of the plane.
Fuzzy clustering based parameter clustering method in mandarin speech recognition

In a large vocabulary continuous speech recognition system, to efficiently decrease parameter size and improve the robustness of parameter training, a parameter clustering method by fuzzy clustering is proposed. Based on the structure of the phonetic decision tree, leaf nodes are used for Gaussian clustering and root nodes or shallow leaf nodes are used for covariance sharing. Experimental results show that when the number of Gaussians is reduced by 50%, recognition accuracy only decreases by 0.55%. By combining fuzzy covariance sharing, a total of 4.16% in recognition increase is achieved over the conventional system with approximately the same parameter size.
A SCTP based OAM system of 3G cellular network

OAM (operations, administration and maintenance) system is a very important component of the 3G cellular network. In order to acquire overall management, fast response and steady operation, a SCTP (stream control transmission protocol) based OAM (SOAM) system is proposed. The SOAM system implements the new characters of SCTP such as multi-stream, enforced SACK and heartbeat mechanism on its transport layer, which helps to decrease the message transmission delay and accelerate the link failure detection. Moreover, a new component named SOAM agent is introduced to improve the operation efficiency of the SOAM system. Test results prove that SOAM system achieves better performance on signaling transmission compared with conventional TCP based OAM system
Combining Classifiers in Software Quality Prediction: A Neural Network Approach

Software quality prediction models seek to predict quality factors such as whether a component is fault prone or not. This can be treated as a kind of pattern recognition problem. In pattern recognition, there is a growing use of multiple classifier combinations with the goal to increase recognition performance. In this paper, we propose a neural network approach to combine multiple classifiers. The combination network consists of two neural networks: a Kohonen self-organization network and a multilayer perceptron network. The multilayer perceptron network is used as Dynamic Selection Network (DSN) and Kohonen self-organization network is served as the final combiner. A case study illustrates our approach and provides the evidence that the combination network with DSN performs better than some other popular combining schemes and the DSN can efficiently improve the performance of the combination network.
An adaptive multi-criteria vertical handoff decision algorithm for radio heterogeneous network

In order to improve the accuracy of the vertical handoff decision for radio heterogeneous network, this paper proposes an adaptive multi-criteria vertical handoff (AMVHO) decision algorithm. This algorithm uses a fuzzy inference system (FIS) and a modified Elman neural network (MENN). The FIS adopts crucial criteria of the vertical handoff as the input variables and makes handoff decision based on the defined rule base. The MENN helps to do the prediction for number-of-users of the after-handoff network, which is a pivotal variable of the FIS. Simulation results show that compared with the conventional method, the AMVHO decision algorithm achieves better performance in guaranteeing the quality of service (QoS) of the after-handoff communication.
A novel method for rapid speaker adaptation based on support speaker weighting

In this paper, we propose a novel model-based speaker adaptation method called support speaker weighting (SSW), which performs the adaptation scheme of model combination based on the selected speakers. These speakers, who are acoustically close to the test speaker, are selected from reference speakers using support vector machines (SVM). Compared with GMM/HMM based speaker selection methods, the proposed method can quickly obtain a more optimal speaker subset because the selection is dynamically determined according to the distribution of reference speakers around the test. Experimental results for a large-vocabulary task given in this paper show that this method is both cheaper in terms of memory and more effective than reference speaker weighting (RSW) for tiny amounts of adaptation data. Relative error rate reduction of 4.1% is achieved when only one adaptation sentence is available.
Fuzzy parameter clustering method in speech recognition

The paper proposes a parameter clustering method using fuzzy clustering analysis to decrease the parameter size efficiently and improve the robustness of parameter training in large vocabulary continuous speech recognition systems. Based on the structure of a phonetic decision tree, leaf nodes are used for Gaussian fuzzy clustering and the root node or shallow leaf nodes are used for covariance fuzzy sharing. Experimental results show that, when the number of Gaussians is reduced by 50%, recognition accuracy only decreases by 0.55% compared to the baseline. By combining covariance fuzzy sharing, a significant performance increasing is achieved over the conventional system with approximately the same parameter size.
Restructuring HMM states for speaker adaptation in Mandarin speech recognition

With the tendency of posterior probability taken into account, a state-restructuring method is proposed based on confusions between HMM states. In the method, HMM state is restructured by sharing Gaussian components with its related states and the re-estimation of the increased-parameters, i.e., the inter-state weights, is derived under the EM framework. Experiments are performed on speaker-independent large vocabulary continuous Mandarin speech recognition. The results show the state-restructured systems outperform the baseline system and the combining with MLLR adaptation can lead to consistent and significant improvement on recognition accuracy over MLLR.
FCM BP based parameter clustering method in speech recognition

To efficiently decrease the parameter size and improve the robustness of parameter training, a parameter clustering method based on FCM BP fuzzy clustering analysis is proposed. Based on the structure of phonetic decision tree in state tying, leaf nodes are used for Gaussian clustering and root node or temporary parent nodes are used for covariance sharing. The experimental results show when the number of Gaussians is reduced by 50%, the recognition rate only decreases by 0.55%. By combining covariance sharing, a total of 4.16% recognition increasing is achieved over the conventional system with approximately the same parameter size.
Methods for improving robustness of decision tree in Mandarin speech recognition

Phonetic decision tree based state tying has been widely used in most large vocabulary continuous speech recognition (LVCSR) systems. However, in most cases, the samples of different leaf nodes are very unbalanced, which may affect the recognition performance. In This work, node merging techniques are proposed to alleviate the problem and further decrease the number of senones. On the other hand, in order to lessen the impact of rare triphones on the quality of the decision tree based state tying and improve the accuracy of every final senone, two methods of dealing with rare triphones are added to hidden Markov model (HMM) acoustic modeling before state tying. Experimental results show that these methods greatly improve the robustness of the decision tree and can achieve better performance with even fewer parameters.
A novel inter-system handoff scheme in wireless overlay networks

The wireless overlay network will play an important role in future mobile communications and Internet access. It will be an IP-based network, which includes various wireless access methods and provides all kinds of services such as voice, data and multimedia. Handoff is a critical issue of the wireless overlay network, because it can guarantee seamless roaming among different networks. Based on a WLAN-GPRS wireless overlay network model, and considering channel characteristics, service types and factors influencing the handoff process, a novel inter-system handoff scheme is proposed and relative decision benchmarks are discussed. Simulations on the key parameters, such as throughput, packet loss and handoff delay, prove that the proposed handoff scheme can provide powerful support for seamless handoff and continuous communication.
OOV rejection algorithm based on class-fusion support vector machine for speech recognition

Support vector machine (SVM) is a promising pattern classification technique that implements the structural risk minimization principle (SRM) in statistical learning theory. This paper proposes a new improved SVM, called class fusion support vector machine or FSVM, which has more robustness to noise and outliers than the standard SVM. We present an investigation into the application of FSVM to the out-of-vocabulary (OOV) rejection problem in a DTW based real-time ASR system. The feature vector consisting of parameters such as normalized N-best word scores and their 1st differences are directly derived from the recognition results as input to the OOV rejection process. The performance of the proposed FSVM classifier is compared with the standard SVM and neural networks.