nils-herrmann/clean_scirepeval huggingface dataset:
Intelligent Instruments: Discovering How to Turn Spectral Data into Information

The Optical Plume Anomaly Detection (OPAD) program at the NASA Marshall Space Flight Center is using plume spectroscopy for the diagnosis of the Space Shuttle Main Engines. A challenging part of this program is matching high resolution spectral data with a physicist's model of spectroscopy to produce estimates of metallic erosion through the plume. This paper describes the discovery process used in doing this. The physicist's model had to be debugged in order to discover the various instrument characteristics, discover critical elements of the data, and in general perform exploratory analysis to understand the instrument and the data it produces. This model gives us strong prior knowledge, however, this needs to be incorporated with care. We had to use a range of statistical techniques in our analysis, including one-dimensional super-resolution to determine the instrument response function. The paper concludes with a discussion of the role of discovery in building intelligent instruments that turn real-time data into useful information.
Learning In Networks

Wray L. BuntineThe Research Institute for Advanced Computer Science is operated by Universities Space ResearchAssociation, The American City Building, Suite 212, Columbia, MD 21044 (410) 730-2656Work reported herein was supported by NASA via Contract NAS 2-13721 between NASA and theUniversities Space Research Association (USRA). Work performed at the Research Institute for AdvancedComputer Science (RIACS), NASA Ames Research Center, Moffett Field, CA 94035-1000
Structured Prediction via the Extragradient Method

We present a simple and scalable algorithm for large-margin estimation of structured models, including an important class of Markov networks and combinatorial models. We formulate the estimation problem as a convex-concave saddle-point problem and apply the extragradient method, yielding an algorithm with linear convergence using simple gradient and projection calculations. The projection step can be solved using combinatorial algorithms for min-cost quadratic flow. This makes the approach an efficient alternative to formulations based on reductions to a quadratic program (QP). We present experiments on two very different structured prediction tasks: 3D image segmentation and word alignment, illustrating the favorable scaling properties of our algorithm.
Large margin methods for structured classification : Exponentiated gradient algorithms and PAC-Bayesian generalization bounds

We consider the problem of structured classification , where the task is to predict a label y from an inputx, andy has meaningful internal structure. Our framework includes supervised training of both Markov random fields and weighted context-free grammars as special cases. We describe an algorithm that solves the large-margin optimization problem defined in , using an exponentialfamily (Gibbs distribution) representation of structured objects. The algorithm is efficient – even in cases where the number of labels y i exponential in size – provided that certain expectations under Gibbs distributions can be calculated efficiently. The optimization method we use for structured labels relies on a more general result, specifically the application of exponentiated gradient (EG) updates [4, 5] to quadratic programs (QPs). We describe a new method for solving QPs based on these techniques, and give bounds on its rate of convergence. In addition to their application to the structured-labels task, the EG updates lead to simple algorithms for optimizing “conventional” binary or multiclass SVM problems. Finally, we give a new generalization bound for structured classification, using PAC-Bayesian methods for the analysis of large margin classifiers.
Exponentiated Gradient Algorithms for Large-margin Structured Classification

We consider the problem of structured classification, where the task is to predict a label y from an input x, and y has meaningful internal structure. Our framework includes supervised training of Markov random fields and weighted context-free grammars as special cases. We describe an algorithm that solves the large-margin optimization problem defined in , using an exponential-family (Gibbs distribution) representation of structured objects. The algorithm is efficient—even in cases where the number of labels y is exponential in size—provided that certain expectations under Gibbs distributions can be calculated efficiently. The method for structured labels relies on a more general result, specifically the application of exponentiated gradient updates [7, 8] to quadratic programs.
1 Spectral Methods for Dimensionality Reduction

How can we search for low dimensional structure in high dimensional data? If the data is mainly confined to a low dimensional subspace, then simple linear methods can be used to discover the subspace and estimate its dimensionality. More generally, though, if the data lies on (or near) a low dimensional submanifold, then its structure may be highly nonlinear, and linear methods are bound to fail. Spectral methods have recently emerged as a powerful tool for nonlinear dimensionality reduction and manifold learning. These methods are able to reveal low dimensional structure in high dimensional data from the top or bottom eigenvectors of specially constructed matrices. To analyze data that lies on a low dimensional submanifold, the matrices are constructed from sparse weighted graphs whose vertices represent input patterns and whose edges indicate neighborhood relations. The main computations for manifold learning are based on tractable, polynomial-time optimizations, such as shortest path problems, least squares fits, semidefinite programming, and matrix diagonalization. This chapter provides an overview of unsupervised learning algorithms that can be viewed as spectral methods for linear and nonlinear dimensionality reduction.
Analysis and extension of spectral methods for nonlinear dimensionality reduction

Many unsupervised algorithms for nonlinear dimensionality reduction, such as locally linear embedding (LLE) and Laplacian eigenmaps, are derived from the spectral decompositions of sparse matrices. While these algorithms aim to preserve certain proximity relations on average, their embeddings are not explicitly designed to preserve local features such as distances or angles. In this paper, we show how to construct a low dimensional embedding that maximally preserves angles between nearby data points. The embedding is derived from the bottom eigenvectors of LLE and/or Laplacian eigenmaps by solving an additional (but small) problem in semidefinite programming, whose size is independent of the number of data points. The solution obtained by semidefinite programming also yields an estimate of the data's intrinsic dimensionality. Experimental results on several data sets demonstrate the merits of our approach.
Learning a kernel matrix for nonlinear dimensionality reduction

We investigate how to learn a kernel matrix for high dimensional data that lies on or near a low dimensional manifold. Noting that the kernel matrix implicitly maps the data into a nonlinear feature space, we show how to discover a mapping that "unfolds" the underlying manifold from which the data was sampled. The kernel matrix is constructed by maximizing the variance in feature space subject to local constraints that preserve the angles and distances between nearest neighbors. The main optimization involves an instance of semidefinite programming---a fundamentally different computation than previous algorithms for manifold learning, such as Isomap and locally linear embedding. The optimized kernels perform better than polynomial and Gaussian kernels for problems in manifold learning, but worse for problems in large margin classification. We explain these results in terms of the geometric properties of different kernels and comment on various interpretations of other manifold learning algorithms as kernel methods.
Multiband statistical learning for f/sub 0/ estimation in speech

We investigate a simple algorithm that combines multiband processing and least squares fits to estimate f/sub 0/ contours in speech. The algorithm is untraditional in several respects: it makes no use of FFTs or autocorrelation at the pitch period; it updates the pitch incrementally on a sample-by-sample basis; it avoids peak picking and does not require interpolation in time or frequency to obtain high resolution estimates; it works reliably, in real time, without the need for postprocessing to produce smooth contours. We show that a baseline implementation of the algorithm, though already quite accurate, is significantly improved by incorporating a model of statistical learning into its final stages. Model parameters are estimated from training data to minimize the likelihood of gross errors in f/sub 0/, as well as errors in classifying voiced versus unvoiced speech. Experimental results on several databases confirm the benefits of statistical learning.
Real-Time Pitch Determination of One or More Voices by Nonnegative Matrix Factorization

An auditory "scene", composed of overlapping acoustic sources, can be viewed as a complex object whose constituent parts are the individual sources. Pitch is known to be an important cue for auditory scene analysis. In this paper, with the goal of building agents that operate in human environments, we describe a real-time system to identify the presence of one or more voices and compute their pitch. The signal processing in the front end is based on instantaneous frequency estimation, a method for tracking the partials of voiced speech, while the pattern-matching in the back end is based on nonnegative matrix factorization, an unsupervised algorithm for learning the parts of complex objects. While supporting a framework to analyze complicated auditory scenes, our system maintains real-time operability and state-of-the-art performance in clean speech.
Statistical signal processing with nonnegativity constraints

Nonnegativity constraints arise frequently in statistical learning and pattern recognition. Multiplicative updates provide natural solutions to optimizations involving these constraints. One well known set of multiplicative updates is given by the Expectation-Maximization algorithm for hidden Markov models, as used in automatic speech recognition. Recently, we have derived similar algorithms for nonnegative deconvolution and nonnegative quadratic programming. These algorithms have applications to low-level problems in voice processing, such as fundamental frequency estimation, as well as high-level problems, such as the training of large margin classifiers. In this paper, we describe these algorithms and the ideas that connect them. Disciplines Computer Engineering | Computer Sciences Comments Proceedings of the Eighth European Conference on Speech Communication and Technology, volume 2, pages 1001-1004, held 1-4 September 2003 in Geneva, Switzerland. This conference paper is available at ScholarlyCommons: http://repository.upenn.edu/cis_papers/13 Statistical Signal Processing with Nonnegativity Constraints Lawrence K. Saul, Fei Sha, Daniel D. Lee Department of Computer and Information Science Department of Electrical and System Engineering University of Pennsylvania, Philadephia, PA {lsaul,feisha,ddlee}@seas.upenn.edu
Shallow Parsing with Conditional Random Fields

Conditional random fields for sequence labeling offer advantages over both generative models like HMMs and classifiers applied at each sequence position. Among sequence labeling tasks in language processing, shallow parsing has received much attention, with the development of standard evaluation datasets and extensive comparison among methods. We show here how to train a conditional random field to achieve performance as good as any reported base noun-phrase chunking method on the CoNLL task, and better than any reported single model. Improved training methods based on modern optimization algorithms were critical in achieving these results. We present extensive comparisons between models and training methods that confirm and strengthen previous results on shallow parsing and training methods for maximum-entropy models.
Multiplicative Updates for Large Margin Classifiers

Various problems in nonnegative quadratic programming arise in the training of large margin classifiers. We derive multiplicative updates for these problems that converge monotonically to the desired solutions for hard and soft margin classifiers. The updates differ strikingly in form from other multiplicative updates used in machine learning. In this paper, we provide complete proofs of convergence for these updates and extend previous work to incorporate sum and box constraints in addition to nonnegativity.
Multiplicative Updates for Nonnegative Quadratic Programming in Support Vector Machines

We derive multiplicative updates for solving the nonnegative quadratic programming problem in support vector machines (SVMs). The updates have a simple closed form, and we prove that they converge monotonically to the solution of the maximum margin hyperplane. The updates optimize the traditionally proposed objective function for SVMs. They do not involve any heuristics such as choosing a learning rate or deciding which variables to update at each iteration. They can be used to adjust all the quadratic programming variables in parallel with a guarantee of improvement at each iteration. We analyze the asymptotic convergence of the updates and show that the coefficients of non-support vectors decay geometrically to zero at a rate that depends on their margins. In practice, the updates converge very rapidly to good classifiers.
The EM algorithm in independent component analysis

We investigate two techniques for independent component analysis which use the expectation-maximization algorithm. Analysis and simulations show that convergence becomes extraordinarily slow for almost all cases, compared to other optimization techniques. The two alternatives considered are "adaptive overrelaxed EM" and Ucminf (a BFGS with soft line search), which both improves the convergence dramatically with little or no extra analytical work. We discuss the generality and perspectives of the findings.
On the Slow Convergence of EM and VBEM in Low-Noise Linear Models

We analyze convergence of the expectation maximization (EM) and variational Bayes EM (VBEM) schemes for parameter estimation in noisy linear models. The analysis shows that both schemes are inefficient in the low-noise limit. The linear model with additive noise includes as special cases independent component analysis, probabilistic principal component analysis, factor analysis, and Kalman filtering. Hence, the results are relevant for many practical applications.
THE EM ALGORITHM IN INDEPENDENT COMPONENT ANALYSIS Kaare

We investigate two techniques for Independent Component Analysis which use the Expectation-Maximization algorithm. Analysis and simulations show that convergence becomes extraordinary slow for almost all cases, compared to other optimization techniques. The two alternatives considered are ”Adaptive Overrelaxed EM” and Ucminf (a BFGS with soft line search), which both improves the convergence dramatically with little or no extra analytical work. We discuss the generality and perspectives of the findings.
FRAME SELECTION FOR SPEAKER IDENTIFICATION

A novel approach to automatic speaker identification is presented. Using low-level acoustic feature sets, a frame based analysis of the system performance is implemented to locate the areas of a speech signal that contain a high level of speaker dependent information. Subgroups of frames are used for the text independent speaker identification task and the resulting system performance is compared with that of using all available frames. It is found that by exclusively using the frames in the transient areas of a speech signal, where the signal shifts between being voiced and unvoiced, the rate of correctly classified frames is increased by up to 14% compared to the case of using randomly selected frames. These results are obtained for PLPCC feature sets extracted from clean speech.
Expectation Consistent Approximate Inference

We propose a novel framework for approximations to intractable probabilistic models which is based on a free energy formulation. The approximation can be understood as replacing an average over the original intractable distribution with a tractable one. It requires two tractable probability distributions which are made consistent on a set of moments and encode different features of the original intractable distribution. In this way we are able to use Gaussian approximations for models with discrete or bounded variables which allow us to include non-trivial correlations. These are neglected in many other methods. We test the framework on toy benchmark problems for binary variables on fully connected graphs and 2D grids and compare with other methods, such as loopy belief propagation. Good performance is already achieved by using single nodes as tractable substructures. Significant improvements are obtained when a spanning tree is used instead.
Expectation Consistent Free Energies for Approximate Inference

We propose a novel a framework for deriving approximations for intractable probabilistic models. This framework is based on a free energy (negative log marginal likelihood) and can be seen as a generalization of adaptive TAP [1, 2, 3] and expectation propagation (EP) [4, 5]. The free energy is constructed from two approximating distributions which encode different aspects of the intractable model such a single node constraints and couplings and are by construction consistent on a chosen set of moments. We test the framework on a difficult benchmark problem with binary variables on fully connected graphs and 2D grid graphs. We find good performance using sets of moments which either specify factorized nodes or a spanning tree on the nodes (structured approximation). Surprisingly, the Bethe approximation gives very inferior results even on grids.
Low complexity Bayesian single channel source separation

We propose a simple Bayesian model for performing single channel speech separation using factorized source priors in a sliding window linearly transformed domain. Using a one dimensional mixture of Gaussians to model each band source leads to fast tractable inference for the source signals. Simulations with separation of a male and a female speaker using priors trained on the same speakers show comparable performance with the blind separation approach of G.-J. Jang and T.-W. Lee (see NIPS, vol.15, 2003) with a SNR improvement of 4.9 dB for both the male and female speaker. Mixing coefficients can be estimated quite precisely using ML-II, but the estimation is quite sensitive to the accuracy of the priors as opposed to the source separation quality for known mixing coefficients, which is quite insensitive to the accuracy of the priors. Finally, we discuss how to improve our approach while keeping the complexity low using machine learning and CASA (computational auditory scene analysis) approaches (Jang and Lee, 2003; Roweis, S.T., 2001; Wang, D.L. and Brown, G.J., 1999; Hu, G. and Wang, D., 2003).
ImprovedMultistageDetectorby Mean-FieldAnnealing in Multi-UserCDMA

We proposea multistage multi-user detector for Code Division Multiple Accessthat implements Mean-Field Annealing. Annealing is a well known technique to avoid local minima in detection and optimisation problems. Local minima certainly occurs in CDMA multi-user detection leading to anomalousBit Err or Rate curves. Annealing is implementedasa standard multi-user detector implementing interfer encecancellation,but with a control parameter determining the slopeof the squashing/mappingfunction. We deri ve a bound on the control parameter for the first stageof the detector. We alsoproposean annealingscheme.Weshow empirically the performanceof the proposeddetector, which follows closelythe optimum detector implementedby exhaustive search, i.e. has gains of several dB’s compared to standard multiuser detectors lik e linear-MMSE, linear-MMSE first stagefollowed by interferencecancellation,and multi-stage interfer encecancellationwith clipped soft decision,the lastbeingthe closestto our proposeddetector. Theproposeddetector gainsup to dB compared to the Clipped Soft DecisionMultistage detector at no addedcomplexity. CDMA, Multistage Multi-User Detection, Interfer enceCancellation, Anomalous -curves.
Independent component analysis for understanding multimedia content

Independent component analysis of combined text and image data from Web pages has potential for search and retrieval applications by providing more meaningful and context dependent content. It is demonstrated that ICA of combined text and image features has a synergistic effect, i.e., the retrieval classification rates increase if based on multimedia components relative to single media analysis. For this purpose a simple probabilistic supervised classifier which works from unsupervised ICA features is invoked. In addition, we demonstrate the suggested framework for automatic annotation of descriptive key words to images.
Mean-Field Approaches to Independent Component Analysis

We develop mean-field approaches for probabilistic independent component analysis (ICA). The sources are estimated from the mean of their posterior distribution and the mixing matrix (and noise level) is estimated by maximum a posteriori (MAP). The latter requires the computation of (a good approximation to) the correlations between sources. For this purpose, we investigate three increasingly advanced mean-field methods: the variational (also known as naive mean field) approach, linear response corrections, and an adaptive version of the Thouless, Anderson and Palmer (1977) (TAP) mean-field approach, which is due to Opper and Winther (2001). The resulting algorithms are tested on a number of problems. On synthetic data, the advanced mean-field approaches are able to recover the correct mixing matrix in cases where the variational mean-field theory fails. For handwritten digits, sparse encoding is achieved using nonnegative source and mixing priors. For speech, the mean-field method is able to separate in the underdetermined (overcomplete) case of two sensors and three sources. One major advantage of the proposed method is its generality and algorithmic simplicity. Finally, we point out several possible extensions of the approaches developed here.
Incremental Gaussian Processes

In this paper, we consider Tipping's relevance vector machine (RVM)  and formalize an incremental training strategy as a variant of the expectation-maximization (EM) algorithm that we call Subspace EM (SSEM). Working with a subset of active basis functions, the sparsity of the RVM solution will ensure that the number of basis functions and thereby the computational complexity is kept low. We also introduce a mean field approach to the intractable classification model that is expected to give a very good approximation to exact Bayesian inference and contains the Laplace approximation as a special case. We test the algorithms on two large data sets with O(103 - 104) examples. The results indicate that Bayesian learning of large data sets, e.g. the MNIST database is realistic.