nils-herrmann/clean_scirepeval huggingface dataset:
Fingerprinting to Identify Repeated Sound Events in Long-Duration Personal Audio Recordings

Body-worn solid-state audio recorders can easily and cheaply capture the bearer's entire acoustic environment throughout the day; we refer to such recordings as "personal audio". Extracting useful information, and providing access and navigation tools for this data is a challenge; in this paper we investigate the use of an audio fingerprinting technique, originally developed for identifying music recordings corrupted by noise, as a tool to rapidly identify recurrent sound events within long (multi-day) recordings. The fingerprinting technique is based on energy peaks in time-frequency, largely removing framing issues and making it intrinsically robust to background noise levels. We show that the technique is very effective at identifying exact repetitions of structured sound (such as jingles and electronic telephone rings) but is unable to find repeats of more 'organic' sound events such as garage door openings.
A computer implementation of psychoacoustic grouping rules

We are building a computer model of sound organization and understanding in human listeners. In particular, we would like to be able to detect and locate acoustic events that will be perceived as separate objects. Our model aims to duplicate this aspect of the auditory system although the level of correspondence is speculative given our current state of knowledge. We describe an implementation of grouping rules corresponding to the psychoacoustic cues of harmonicity, common onset, continuity and proximity. We increase the system's robustness by adding a second layer of grouping that looks for corroboration between primary groupings. We believe that such a system of repeated hierarchic grouping is critical for the successful modeling of auditory functions.
Structured Prediction Models for Chord Transcription of Music Audio

Chord sequences are a compact and useful description of music, representing each beat or measure in terms of a likely distribution over individual notes without specifying the notes exactly. Transcribing music audio into chord sequences is essential for harmonic analysis, and would be an important component in content-based retrieval and indexing, but accuracy rates remain fairly low. In this paper, the existing 2008 LabROSA Supervised Chord Recognition System is modified by using different machine learning methods for decoding structural information, thereby achieving significantly superior results. Specifically, the hidden Markov model is replaced by a large margin structured prediction approach (SVMstruct) using an enlarged feature space. Performance is significantly improved by incorporating features from future (but not past) frames. The benefit of SVMstruct increases with the size of the training set, as might be expected when comparing discriminative and generative models. Without yet exploring non-linear kernels, these improvements lead to state-of-the-art performance in chord transcription. The techniques could prove useful in other sequential learning tasks which currently employ HMMs.
Optimizing DTW-based audio-to-MIDI alignment and matching

Dynamic time warping (DTW) has proven to be an extremely effective method for both aligning and matching recordings of music to corresponding MIDI transcriptions. However, its performance is heavily affected by factors such as the representation used for the audio and MIDI data and its adjustable parameters. We therefore investigate automatically optimizing the design of DTW-based alignment and matching systems. Our approach uses Bayesian optimization to tune system design and parameters over a synthetically-created dataset of audio and MIDI pairs. We then perform an exhaustive search over DTW score normalization techniques to find the optimal method for reporting a reliable alignment confidence score, as required in matching tasks. This results in a DTW-based system which is conceptually simple and highly accurate at both alignment and matching. We verified that this system achieves high performance in a large-scale qualitative evaluation of real-world alignments.
Speech separation in humans and machines

Any system for extracting information from sound must face the problem of interference between multiple overlapping sound sources. The human auditory system does this very successfully, but our computer systems have some way to go. In this talk, I would review various approaches to separating information from sound mixtures, particularly for the difficult case of target voice mixed with interfering voices. I also review some results in human abilities to understand sound mixtures, and what these may be able to tell us about promising automatic approaches
Speech decoloration based on the product-of-filters model

We present a single-channel speech decoloration method based on a recently proposed generative product-of-filters (PoF) model. We take a spectral approach and attempt to learn the magnitude response of the actual coloration filter, given only the degraded speech signal. Experiments on synthetic data demonstrate that the proposed method effectively captures both coarse and fine structure of the coloration filter. On real recordings, we find that simply subtracting the learned coloration filter from the log-spectra yields promising decoloration results.
Underconstrained stochastic representations for top-down computational auditory scene analysis

I propose a structure for the first stage of a computer system capable of performing complex auditory scene analysis similar to that accomplished by human listeners. This structure contains the following innovations over previous approaches: (1) Sound is represented as discrete elements drawn from an overcomplete vocabulary encompassing both tonal and less structured sounds, designed to highlight the interdependence in the acoustic energy. (2) Through the redundancy of the basis this analysis permits and indeed requires the imposition of additional constraints, which provides for the incorporation of top-down or context-sensitive factors. (3) A modular architecture operates on an analysis-by-synthesis principle, where processes are invoked until the representation adequately accounts for the observed sound. A common goodness-of-fit criterion allows for future expansion of the system with new explanation rules, new representational elements and more abstract levels of analysis. Some initial results of applying these ideas to scenes consisting of noise bursts and dense environmental sound are presented.
Detecting music in ambient audio by long-window autocorrelation

We address the problem of detecting music in the background of ambient real-world audio recordings such as the sound track of consumer-shot video. Such material may contain high levels of noises, and we seek to devise features that will reveal music content in such circumstances. Sustained, steady musical pitches show significant, structured autocorrelation at when calculated over windows of hundreds of milliseconds, where autocorrelation of aperiodic noise has become negligible at higher-lag points if a signal is whitened by LPC. Using such features, further compensated by their long-term average to remove the effect of stationary periodic noise, we produce GMM and SVM based classifiers with high performance compared with previous approaches, as verified on a corpus of real consumer video.
CNN architectures for large-scale audio classification

Convolutional Neural Networks (CNNs) have proven very effective in image classification and show promise for audio. We use various CNN architectures to classify the soundtracks of a dataset of 70M training videos (5.24 million hours) with 30,871 video-level labels. We examine fully connected Deep Neural Networks (DNNs), AlexNet , VGG , Inception , and ResNet . We investigate varying the size of both training set and label vocabulary, finding that analogs of the CNNs used in image classification do well on our audio classification task, and larger training and label sets help up to a point. A model using embeddings from these classifiers does much better than raw features on the Audio Set  Acoustic Event Detection (AED) classification task.
Pitch-based emphasis detection for characterization of meeting recordings

The automatic extraction of key utterances in spoken data has emerged as an interesting and difficult topic in automatic speech recognition. "Emphasis" or "excitement" may be a useful identifier for these utterances of interest. We undertake the task of reliably and automatically identifying emphasized or excited utterances in natural speech in a meeting setting. We start by endeavoring to establish reliable ground truth emphasis labels by using several hand-labelers. The results show that human listeners can reliably identify emphasized utterances in meeting recordings. We then build an automatic emphasis detection system, which uses normalized pitch as its only acoustic predictor. The results show that this pitch-based emphasis detection scheme can distinguish between non-emphasized and emphasized utterances with an accuracy of 92% when ambiguous cases are excluded, a rate comparable to human interlabeler agreement.
Hierarchic models of hearing for sound separation and reconstruction

In building a machine to detect and segregate individual components in sound mixtures, the best example to copy is the human auditory system. Several models of auditory organization implement various rules of psychoacoustic grouping. We propose in addition to model auditory inference as exhibited in the well-known 'phonemic restoration illusion' of Warren (1970). A hierarchy of abstracted features and source hypotheses similar to that of Nawab (1992) allows reconstruction of obliterated detail which can then be used to recreate an 'idealized' sound without corruption. A preliminary example of fitting a harmonic model to a noisy recording of a clarinet gives a very convincing resynthesis with the interference totally removed. However, there are many issues including the design of the representation and the control architecture still to be addressed in building a more general system.<<ETX>>
Transcribing Multi-Instrument Polyphonic Music With Hierarchical Eigeninstruments

This paper presents a general probabilistic model for transcribing single-channel music recordings containing multiple polyphonic instrument sources. The system requires no prior knowledge of the instruments present in the mixture (other than the number), although it can benefit from information about instrument type if available. In contrast to many existing polyphonic transcription systems, our approach explicitly models the individual instruments and is thereby able to assign detected notes to their respective sources. We use training instruments to learn a set of linear manifolds in model parameter space which are then used during transcription to constrain the properties of models fit to the target mixture. This leads to a hierarchical mixture-of-subspaces design which makes it possible to supply the system with prior knowledge at different levels of abstraction. The proposed technique is evaluated on both recorded and synthesized mixtures containing two, three, four, and five instruments each. We compare our approach in terms of transcription with (i.e., detected pitches must be associated with the correct instrument) and without source-assignment to another multi-instrument transcription system as well as a baseline non-negative matrix factorization (NMF) algorithm. For two-instrument mixtures evaluated with source-assignment, we obtain average frame-level F-measures of up to 0.52 in the completely blind transcription setting (i.e., no prior knowledge of the instruments in the mixture) and up to 0.67 if we assume knowledge of the basic instrument types. For transcription without source assignment, these numbers rise to 0.76 and 0.83, respectively.
Cross-correlation of beat-synchronous representations for music similarity

Systems to predict human judgments of music similarity directly from the audio have generally been based on the global statistics of spectral feature vectors i.e. collapsing any large-scale temporal structure in the data. Based on our work in identifying alternative ("cover") versions of pieces, we investigate using direct correlation of beat-synchronous representations of music audio to find segments that are similar not only in feature statistics, but in the relative positioning of those features in tempo-normalized time. Given a large enough search database, good matches by this metric should have very high perceived similarity to query items. We evaluate our system through a listening test in which subjects rated system-generated matches as similar or not similar, and compared results to a more conventional timbral and rhythmic similarity baseline, and to random selections.
Modelling Sound Dynamics Using Deformable Spectrograms: Segmenting the Spectrogram into Smooth Regions

Speech and other natural sounds show high temporal correlation and smooth spectral evolution punctuated by a few, irregular and abrupt changes. We model successive spectra as transformations of their immediate predecessors, capturing the evolution of the signal energy through time. The speech production model is used to decomposed the log-spectrogrum into two additive layers, which are able to separately explain and model the evolution of the harmonic excitation, and formant filtering of speech and similar sounds. We present results on a speech recognition task, that suggest that the model discovers a global structure on the dynamics of the signal's energy that helps to alleviate the problems generated by noise interferences. The model is also used to segment mixtures of speech into dominant speaker regions on a unsupervised source separation task.
Active Learning for Interactive Multimedia Retrieval

As the first decade of the 21st century comes to a close, growth in multimedia delivery infrastructure and public demand for applications built on this backbone are converging like never before. The push towards reaching truly interactive multimedia technologies becomes stronger as our media consumption paradigms continue to change. In this paper, we profile a technology leading the way in this revolution: active learning. Active learning is a strategy that helps alleviate challenges inherent in multimedia information retrieval through user interaction. We show how active learning is ideally suited for the multimedia information retrieval problem by giving an overview of the paradigm and component technologies used with special attention given to the application scenarios in which these technologies are useful. Finally, we give insight into the future of this growing field and how it fits into the larger context of multimedia information retrieval.
Leveraging repetition for improved automatic lyric transcription in popular music

Transcribing lyrics from musical audio is a challenging research problem which has not benefited from many advances made in the related field of automatic speech recognition, owing to the prevalent musical accompaniment and differences between the spoken and sung voice. However, one aspect of this problem which has yet to be exploited by researchers is that significant portions of the lyrics will be repeated throughout the song. In this paper we investigate how this information can be leveraged to form a consensus transcription with improved consistency and accuracy. Our results show that improvements can be gained using a variety of techniques, and that relative gains are largest under the most challenging and realistic experimental conditions.
Learning to segment songs with ordinal linear discriminant analysis

This paper describes a supervised learning algorithm which optimizes a feature representation for temporally constrained clustering. The proposed method is applied to music segmentation, in which a song is partitioned into functional or locally homogeneous segments (e.g., verse or chorus). To facilitate abstraction over multiple training examples, we develop a latent structural repetition feature, which summarizes the repetitive structure of a song of any length in a fixed-dimensional representation. Experimental results demonstrate that the proposed method efficiently integrates heterogeneous features, and improves segmentation accuracy.
The ICSI Meeting Corpus

We have collected a corpus of data from natural meetings that occurred at the International Computer Science Institute (ICSI) in Berkeley, California over the last three years. The corpus contains audio recorded simultaneously from head-worn and table-top microphones, word-level transcripts of meetings, and various metadata on participants, meetings, and hardware. Such a corpus supports work in automatic speech recognition, noise robustness, dialog modeling, prosody, rich transcription, information retrieval, and more. We present details on the contents of the corpus, as well as rationales for the decisions that led to its configuration. The corpus were delivered to the Linguistic Data Consortium (LDC).
Stylization of pitch with syllable-based linear segments

Fundamental frequency contours for speech, as obtained by common pitch tracking algorithms, contain a great deal of fine detail that is unlikely to hold much perceptual significance for listeners. In our experiments, a radically reduced pitch contour consisting of a single linear segment for each syllable was found to judged as equally natural as the original pitch track by listeners, based on high-quality analysis- synthesis. We describe the algorithms both for segmenting speech into syllables based on fitting Gaussians to the energy envelope, and for approximating the pitch contour by independent linear segments for each syllable. We report our web-based test in which 40 listeners compared the stylized pitch contour resyntheses to equivalent resyntheses based on the original pitch track, and also to pitch tracks stylized by the existing Momel algorithm. Listeners preferred the original pitch contour to the linear approximation in only 60% of cases, where 50% would indicate random guessing. By contrast, the original was preferred over Momel in 74% of cases.
Better beat tracking through robust onset aggregation

Onset detection forms the critical first stage of most beat tracking algorithms. While common spectral-difference onset detectors can work well in genres with clear rhythmic structure, they can be sensitive to loud, asynchronous events (e.g., off-beat notes in a jazz solo), which limits their general efficacy. In this paper, we investigate methods to improve the robustness of onset detection for beat tracking. Experimental results indicate that simple modifications to onset detection can produce large improvements in beat tracking accuracy.
Modeling nonlinear circuits with linearized dynamical models via kernel regression

This paper introduces a novel method for the solution of guitar distortion circuits based on the use of kernels. The proposed algorithm uses a kernel regression framework to linearize the inherent nonlinear dynamical systems created by such circuits and proposes data and kernel selection algorithms well suited to learn the required regression parameters. Examples are presented using the One Capacitor Diode Clipper and the Common-Cathode Tube Amplifier.
Evaluating music sequence models through missing data

Building models of the structure in musical signals raises the question of how to evaluate and compare different modeling approaches. One possibility is to use the model to impute deliberately-removed patches of missing data, then to compare the model's predictions with the part that was removed. We analyze a corpus of popular music audio represented as beat-synchronous chroma features, and compare imputation based on simple linear prediction to more complex models including nearest neighbor selection and shift-invariant probabilistic latent component analysis. Simple linear models perform best according to Euclidean distance, despite producing stationary results which are not musically meaningful. We therefore investigate alternate evaluation measures and observe that an entropy difference metric correlates better with our expectations for musically consistent reconstructions. Under this measure, the best-performing imputation algorithm reconstructs masked sections by choosing the nearest neighbor to the surrounding observations within the song. This result is consistent with the large amount of repetition found in pop music.
Visual saliency based on fast nonparametric multidimensional entropy estimation

Bottom-up visual saliency can be computed through information theoretic models but existing methods face significant computational challenges. Whilst nonparametric methods suffer from the curse of dimensionality problem and are computationally expensive, parametric approaches have the difficulty of determining the shape parameters of the distribution models. This paper makes two contributions to information theoretic based visual saliency models. First, we formulate visual saliency as center surround conditional entropy which gives a direct and intuitive interpretation of the center surround mechanism under the information theoretic framework. Second, and more importantly, we introduce a fast nonparametric multidimensional entropy estimation solution to make information theoretic-based saliency models computationally tractable and practicable in realtime applications. We present experimental results on publicly available eye-tracking image databases to demonstrate that the proposed method is competitive to state of the art.
Colour-based bottom-up saliency for traffic sign detection

On roads, drivers can detect traffic sign extraordinarily fast and accurately; however, the computer vision system does not easily imitate this natural ability although using a lot of image processing techniques. This weakness can be tackled by a bottom-up visual saliency method based on traffic-sign colours which occupy certain ranges of RGB values. These values can be used to modulate the bottom-up visual saliency, so the proposed system can focus on traffic signs detection. The method is tested on the UNMC Automotive Vision Database and compared with results of the purely bottom-up visual saliency method.
Constraint adaptive segmentation for color image coding and content-based retrieval

We present a constraint adaptive image segmentation technique designed to achieve the combined purposes of color image coding/compression, indexing and content-based retrieval. An image is segmented into homogeneous squared regions of variable sizes such that it can be encoded very efficiently. From the segmented image, we derive an effective and efficient image content description feature termed the region and color co-occurrence matrix (RACOM) as image index for content-based indexing and retrieval. Experimental results are presented to demonstrate that RACOM is a very effective image content description feature and has comparable performance to state of the art methods, colour correlogram and MPEG7 colour structure histogram, in content-based image retrieval from large image databases.
Color photo categorization using compressed histograms and support vector machines

In this paper, an efficient method using various histogram-based (high-dimensional) image content descriptors for automatically classifying general color photos into relevant categories is presented. Principal component analysis (PCA) is used to project the original high dimensional histograms onto their eigenspaces. Lower dimensional eigenfeatures are then used to train support vector machines (SVMs) to classify images into their categories. Experimental results show that even though different descriptors perform differently, they are all highly redundant. It is shown that the dimensionality of all these descriptors, regardless of their performances, can be significantly reduced without affecting classification accuracy. Such scheme would be useful when it is used in an interactive setting for relevant feedback in content-based image retrieval, where low dimensional content descriptors enable fast online learning and reclassification of results.