nils-herrmann/clean_scirepeval huggingface dataset:
Agnostic Learning Nonconvex Function Classes

We consider the sample complexity of agnostic learning with respect to squared loss. It is known that if the function class F used for learning is convex then one can obtain better sample complexity bounds than usual. It has been claimed that there is a lower bound that showed there was an essential gap in the rate. In this paper we show that the lower bound proof has a gap in it. Although we do not provide a definitive answer to its validity. More positively, we show one can obtain "fast" sample complexity bounds for nonconvex F for "most" target conditional expectations. The new bounds depend on the detailed geometry of F, in particular the distance in a certain sense of the target's conditional expectation from the set of nonuniqueness points of the class F.
1 The Structure of Version Space

We investigate the generalisation performance of consistent classifiers, i.e. classifiers that are contained in the so-called version space, both from a theoretical and experimental angle. In contrast to classical VC analysis—where no single classifier within version space is singled out on grounds of a generalisation error bound—the data dependent structural risk minimisation framework suggests that there exists one particular classifier that is to be preferred because it minimises the generalisation error bound. This is usually taken to provide a theoretical justification for learning algorithms such as the well known support vector machine. A reinterpretation of a recent PAC-Bayesian result, however, reveals that given a suitably chosen hypothesis space there exists a large fraction of classifiers with small generalisation error albeit we cannot identify them for a specific learning task. In the particular case of linear classifiers we show that classifiers found by the classical perceptron algorithm have guarantees bounded by the size of version space. These results are complemented with an empirical study for kernel classifiers on the task of handwritten digit recognition which demonstrates that even classifiers with a small margin may exhibit excellent generalisation. In order to perform this analysis we introduce the kernel Gibbs sampler—an algorithm which can be used to sample consistent kernel classifiers.
Learning and generalization: theoretical bounds

Novel 2-oxy-5-isopropylidene-7-hydroxy-9-substituted-2,6-methano-3,4,5,6-tet rahydro-2H-1-benzoxocins are prepared by condensing a 5-(substituted)resorcinol with a 1-alkoxy-4-(1-hydroxy-1-methylethyl)-1,4-cyclohexadiene in the presence of a suitable catalyst. The new benzoxocin derivatives are useful in the synthesis of certain dibenzo[b,d]pyran-9-ones, valuable as anti-anxiety, analgesic, and anti-depressant drugs.
Exploiting sparsity in adaptive filters

This paper studies a class of algorithms called natural gradient (NG) algorithms. The least mean square (LMS) algorithm is derived within the NG framework, and a family of LMS variants that exploit sparsity is derived. This procedure is repeated for other algorithm families, such as the constant modulus algorithm (CMA) and decision-directed (DD) LMS. Mean squared error analysis, stability analysis, and convergence analysis of the family of sparse LMS algorithms are provided, and it is shown that if the system is sparse, then the new algorithms will converge faster for a given total asymptotic MSE. Simulations are provided to confirm the analysis. In addition, Bayesian priors matching the statistics of a database of real channels are given, and algorithms are derived that exploit these priors. Simulations using measured channels are used to show a realistic application of these algorithms.
Constant Directivity Beamforming

Beamforming, or spatial filtering, is one of the simplest methods for discriminating between different signals based on the physical location of the sources. Because speech is a very wideband signal, covering some four octaves, traditional narrowband beamforming techniques are inappropriate for hands-free speech acquisition. One class of broadband beamformers, called constant directivity beam-formers, aim to produce a constant spatial response over a broad frequency range. In this chapter we review such beamformers, and discuss implementation issues related to their use in microphone arrays.
Algorithmic Luckiness

Classical statistical learning theory studies the generalisation performance of machine learning algorithms rather indirectly. One of the main detours is that algorithms are studied in terms of the hypothesis class that they draw their hypotheses from. In this paper, motivated by the luckiness framework of Shawe-Taylor et al. (1998), we study learning algorithms more directly and in a way that allows us to exploit the serendipity of the training sample. The main difference to previous approaches lies in the complexity measure; rather than covering all hypotheses in a given hypothesis space it is only necessary to cover the functions which could have been learned using the fixed learning algorithm. We show how the resulting framework relates to the VC, luckiness and compression frameworks. Finally, we present an application of this framework to the maximum margin algorithm for linear classifiers which results in a bound that exploits the margin, the sparsity of the resultant weight vector, and the degree of clustering of the training data in feature space.
Prior Knowledge and Preferential Structures in Gradient Descent Learning Algorithms

A family of gradient descent algorithms for learning linear functions in an online setting is considered. The family includes the classical LMS algorithm as well as new variants such as the Exponentiated Gradient (EG) algorithm due to Kivinen and Warmuth. The algorithms are based on prior distributions defined on the weight space. Techniques from differential geometry are used to develop the algorithms as gradient descent iterations with respect to the natural gradient in the Riemannian structure induced by the prior distribution. The proposed framework subsumes the notion of "link-functions".
Convergence of exponentiated gradient algorithms

This paper studies three related algorithms: the (traditional) gradient descent (GD) algorithm, the exponentiated gradient algorithm with positive and negative weights (EG/spl plusmn/ algorithm), and the exponentiated gradient algorithm with unnormalized positive and negative weights (EGU/spl plusmn/ algorithm). These algorithms have been previously analyzed using the "mistake-bound framework" in the computational learning theory community. We perform a traditional signal processing analysis in terms of the mean square error. A relationship between the learning rate and the mean squared error (MSE) of predictions is found for the family of algorithms. This is used to compare the performance of the algorithms by choosing learning rates such that they converge to the same steady-state MSE. We demonstrate that if the target weight vector is sparse, the EG/spl plusmn/ algorithm typically converges more quickly than the GD or EGU/spl plusmn/ algorithms that perform very similarly. A side effect of our analysis is a reparametrization of the algorithms that provides insights into their behavior. The general form of the results we obtain are consistent with those obtained in the mistake-bound framework. The application of the algorithms to acoustic echo cancellation is then studied, and it is shown in some circumstances that the EG/spl plusmn/ algorithm will converge faster than the other two algorithms.
Very low voltage power conversion

A DC-DC converter for use with very low input voltages is presented. Operating with inputs as low as 0.3 VDC, the design also incorporates magnetic drive characteristics which significantly reduce the likelihood of core saturation. The circuit was developed to demonstrate the feasibility of efficient and economical electrical energy conversion from parallel arrays of solar cells operating at moderate to high power levels. While the circuit described here has a nominal 24 VDC output, a DC (or AC) output of almost any voltage is equally practical. Converter evaluation necessitated the development of a 500 A adjustable regulated low voltage DC power supply. The design of this supply is also considered.
Riemannian structure of some new gradient descent learning algorithms

We consider some generalizations of the classical LMS learning algorithm including the exponentiated gradient (EG) algorithm. We show how one can develop these algorithms in terms of a prior distribution over the weight space. Our framework subsumes the notion of "link-functions". Differential geometric methods are used to develop the algorithms as gradient descent with respect to the natural gradient in the Riemannian structure induced by the prior distribution. This provides a Bayesian Riemannian interpretation of the EG and related algorithms. We relate our work to that of Amari (1985, 1997, 1998) and others who used similar tools in a different manner. Simulation experiments illustrating the behaviour of the new algorithms are presented.
Equalization in an acoustic reverberant environment: robustness results

This paper investigates the robustness of sound equalization using a room response inverse filter with respect to changing or uncertain source or microphone positions. It is shown that due to the variations of the transfer function from point to point in a room, even small changes in the source or microphone position of just a few tenths of the acoustic wavelength can cause large degradations in the equalized room response. The robustness problem is especially acute at high frequencies, which are known to carry some important attributes of the speech signal. The spatial extent of equalization, derived from the statistical-average properties of sound transmission in rooms, is illustrated by computer simulations which corroborate the theoretical results presented.
From Margin to Sparsity

We present an improvement of Novikoff's perceptron convergence theorem. Reinterpreting this mistake bound as a margin dependent sparsity guarantee allows us to give a PAC-style generalisation error bound for the classifier learned by the perceptron learning algorithm. The bound value crucially depends on the margin a support vector machine would achieve on the same data set using the same kernel. Ironically, the bound yields better guarantees than are currently available for the support vector solution itself.
Error performance of a channel of known impulse response

This paper addresses the problem of blind channel equalization in the context of digital communications. Recent results have shown that certain operations applied to the source signal at the transmitter help in the blind identification and equalization of the channel at the receiver. For example, cyclostationarity is induced by multiplying the baseband data signal with a periodic sequence. Here, the baseband data signal is multiplied with a chirp sequence. Exploiting certain structural properties arising from this operation, a simple and fast converging algorithm is obtained for estimating the equalizer coefficients. The algorithm is deterministic, i.e. it provides optimal equalizer settings using finite data in the noiseless case.
Imposing pattern nulls on broadband array responses

This paper considers the problem of altering a quiescent design for an array of omni-directional sensors so that the altered design rejects a far-field broadband signal from a given direction. This problem occurs where microphone arrays are to be used to acquire speech signals for telecommunication and interferring signals must be rejected. Three main results are presented in this paper. First, conditions for imposing an exact broadband null upon any given quiescent array response are derived. Second, an analysis of the sensitivity of exact nulling array responses to sensor position error is presented. Third, frequency domain formulations for broadband nulls are obtained and it is shown that these are less restrictive than the exact null conditions which are imposed in the time domain. A Lagrange multiplier approach is used to impose the null conditions upon a quiescent array response to minimize the square error between the quiescent and nulled responses. A design example is given.
Nearfield broadband adaptive beamforming

A nearfield broadband adaptive beamforming approach based on the modal expansion of the solution to the spherical wave equation is formulated. It provides an efficient parameterization for the nearfield beamforming problem with a single parameter to focus the beamformer to a desired operating radius and a set of modal coefficients to control the angular response. As a consequence, the adaptive beamformer has a frequency invariant response for nearfield sources, decorrelates coherent interferers (thus combating signal cancellation) and has a smaller number of parameters to be adapted compared to a conventional beamformer. A simulation example is presented to demonstrate the use of the proposed method in microphone array applications in speech acquisition systems.
Isotropic noise modelling for nearfield array processing

An exact series representation for a nearfield spherically isotropic noise model is introduced. The methodology uses the spherical harmonics expansion of the wavefield at a sensor to obtain the correlation between two sensors due to the nearfield isotropic noise field. The result is useful in nearfield application of sensor arrays. The proposed noise model can be utilized effectively to apply well established farfield array processing algorithms for nearfield applications. Specifically, any signal processing criterion based on farfield isotropic noise correlation can be reformulated with nearfield noise with this representation. A simple array gain optimization is used to demonstrate the new noise model.
Finite sample properties of linear model identification

In this paper, we consider the finite sample properties of prediction error methods using a quadratic criterion function for system identification. The problem we pose is: How many data points are required to guarantee with high probability that the expected value of the quadratic identification criterion is close to its empirical mean value? The sample sizes are obtained using risk minimization theory which provides uniform probabilistic bounds on the difference between the expected value of the squared prediction error and its empirical mean evaluated on a finite number of data points. The bounds are very general. No assumption is made about the true system belonging to the model class, and the noise sequence is not assumed to be uniformly bounded. Further analysis shows that in order to maintain a given bound on the deviation, the number of data points needed grows no faster than quadratically with the number of parameters for FIR and ARX models.
On the poor robustness of sound equalization in reverberant environments

This paper examines the sensitivity of sound equalization to source or microphone position changes in a reverberant room. It is demonstrated that even small displacements from the reference (equalization) point, of the order of a tenth of the acoustic wavelength, can cause large degradations in the equalized room response. The general theory developed in this paper, which implies that the sound equalization in practical environments may be an ill-posed problem, is verified by the simulation results averaged over different source and microphone positions.
An analysis of the exponentiated gradient descent algorithm

This paper analyses three algorithms previously studied in the computational learning theory community: the gradient descent (GD) algorithm, the exponentiated gradient algorithm with positive and negative weights (EG/spl plusmn/ algorithm) and the exponentiated gradient algorithm with unnormalised positive and negative weights (EGU/spl plusmn/ algorithm). The analysis is of the form used in the signal processing community and is in terms of the mean square error. A relationship between the learning rate and the mean squared error (MSE) of predictions is found for the family of algorithms. Trials involving simulated acoustic echo cancellation are conducted whereby learning rates for the algorithms are selected such that they converge to the same steady state MSE. These trials demonstrate that, in the case that the target is sparse, the EG/spl plusmn/ algorithm typically converges more quickly than the GD or EGU/spl plusmn/ algorithms which perform very similarly.
Generalization Bounds via Eigenvalues of the Gram matrix

Model selection in Support Vector machines is usually carried out by min imizing the quotient of the radius of the smallest enclosing sphere of the data and the observed margin on the training set We provide a new crite rion taking the distribution within that sphere into account by considering the Gram matrix of the data In particular this makes use of the eigen value distribution of the matrix Experimental results on real world data show that this new criterion provides a good prediction of the shape of the curve relating generalization error to kernel width
Generalization Bounds for ConvexCombinations of Kernel

We derive new bounds on covering numbers for hypothesis classes generated by convex combinations of basis functions. These are useful in bounding the generalization performance of algorithms such as RBF-networks, boosting and a new class of linear programming machines similar to SV machines. We show that p-convex combinations with p > 1 lead to diverging bounds, whereas for p = 1 good bounds in terms of entropy numbers can be obtained. In the case of kernel expansions, signiicantly better bounds can be obtained depending on the eigenvalues of the corresponding integral operators.
Generalization Bounds andLearning Rates forRegularized Principal

We derive uniform convergence bounds and learning rates for regularized principal manifolds. This builds on previous work of Kegl et al., however we are able to obtain stronger bounds taking advantage of the decomposition of the principal manifold in terms of kernel functions. In particular, we are able to give bounds on the covering numbers which are independent of the number of basis functions (line elements) used. Finally we are able to obtain a nearly optimal learning rate of order O(m ? 1 2 +) for certain types of regularization operators, where m is the sample size and an arbitrary positive constant. A companion paper 4] describes the basic algorithm, details of the implementation and experimental results.
The Importance of Convexity in Learning with Squared Loss

We show that if the closure of a function class F under the metric induced by some probability distribution is not convex, then the sample complexity for agnostically learning F with squared loss (using only hypotheses in F) is /spl Omega/(ln(1//spl delta/)//spl epsiv//sup 2/) where 1-/spl delta/ is the probability of success and /spl epsiv/ is the required accuracy. In comparison, if the class F is convex and has finite pseudodimension, then the sample complexity is O(1//spl epsiv/(ln(1//spl epsiv/)+ln(1/b)). If a nonconvex class F has finite pseudodimension, then the sample complexity for agnostically learning the closure of the convex hull of F, is O(1//spl epsiv/(1//spl epsiv/(ln(1//spl epsiv/)+ln(1//spl delta/)). Hence, for agnostic learning, learning the convex hull provides better approximation capabilities with little sample complexity penalty.
Broadband beamforming using elementary shape invariant beampatterns

This paper presents a new method of designing a beamformer having a desired broadband beampattern with focusing capability to operate at any radial distance from the array origin. An important consequence of our result is that the beamformer processing can be factored in to three levels of filtering: (i) beampattern independent elementary beamformers; (ii) beampattern shape dependent filters; and (iii) radial zooming filters where a single parameter can be adjusted to zoom-focus the array to a desired radial distance from the array origin. As an illustration the method is applied to the problem of producing a practical array design that achieves a frequency invariant beampattern over the frequency range of 1:10.
Online learning via congregational gradient descent

We propose and analyse a populational version of stepwise gradient descent suitable for a wide range of learning problems. The algorithm is motivated by genetic algorithms which update a population of solutions rather than just a single representative as is typical for gradient descent. This modification of traditional gradient descent (as used, for example, in the backpropogation algorithm) avoids getting trapped in local minima. We use an averaging analysis of the algorithm to relate its behaviour to an associated ordinary differential equation. We derive a result concerning how long one has to wait in order that, with a given high probability, the algorithm is within a certain neighbourhood of the global minimum. We also analyse the effect of different population sizes. An example is presented which corroborates our theory very well.
Decision region approximation by polynomials or neural networks

We give degree of approximation results for decision regions which are defined by polynomial and neural network parametrizations. The volume of the misclassified region is used to measure the approximation error, and results for the degree of L/sub 1/ approximation of functions are used. For polynomial parametrizations, we show that the degree of approximation is at least 1, whereas for neural network parametrizations we prove the slightly weaker result that the degree of approximation is at least r, where r can be any number in the open interval (0, 1).
Correction to Lower Bounds on VC-Dimension of Smoothly Parameterized Function Classes1

The earlier article gives lower bounds on the VC-dimension of various smoothly parameterized function classes. The results were proved by showing a relationship between the uniqueness of decision boundaries and the VC-dimension of smoothly parameterized function classes. The proof is incorrect; there is no such relationship under the conditions stated in the article. For the case of neural networks with tanh activation functions, we give an alternative proof of a lower bound for the VC-dimension proportional to the number of parameters, which holds even when the magnitude of the parameters is restricted to be arbitrarily small.
Broadband Microphone Arrays for Speech Acquisition

A microphone array provides an effective means of hands-free speech acquisition, finding application in large-room videoconferencing, desktop teleconferencing, and hands-free telephones. Unlike a single omni-directional microphone, the array provides high spatial directivity, allowing it to attenuate room reverberation and other unwanted noise. In this paper, we give an overview of the techniques available for speech acquisition using microphone arrays. In particular we will focus on methods of frequency-invariant beamforming, in which the array maintains the same spatial response over a wide frequency range.  Darren Ward Room 2C-545 Bell Labs 600 Mountain Ave Murray Hill, NJ 07974 USA Phone: +1-908-582-5601 Fax: +1-908-582-7308
An adaptive algorithm for broadband frequency invariant beamforming

Frequency invariant beamforming is array processing in which the spatial response remains constant (with respect to frequency) within a wide frequency band of interest. We present a new algorithm for adaptive broadband beamforming which solves a minimum variance beamforming problem, with a structural frequency invariant beampattern constraint. This constraint allows us to reduce the dimension of the adaptation problem. The proposed algorithm is a block adaptive LMS algorithm which uses only a fraction of the parameters of a conventional fully adaptive array. Hence, the computational complexity is reduced and the convergence speed is increased. A simulation example is presented to demonstrate the new algorithm.