nils-herrmann/clean_scirepeval huggingface dataset:
Analysis of multimodal signals using redundant representations

In this work we explore the potentialities of a framework for the representation of audio-visual signals using decompositions on overcomplete dictionaries. Redundant decompositions may describe audio-visual sequences in a concise fashion, preserving good representation properties thanks to the use of redundant, well designed, dictionaries. We expect that this helps us overcome two typical problems of multimodal fusion algorithms. On one hand, classical representation techniques, like pixel-based measures (for the video) or Fourier-like transforms (for the audio), take into account only marginally the physics of the problem. On the other hand, the input signals have large dimensionality. The results we obtain by making use of sparse decompositions of audio-visual signals over redundant codebooks are encouraging and show the potentialities of the proposed approach to multimodal signal representation.
FREAK: Fast Retina Keypoint

A large number of vision applications rely on matching keypoints across images. The last decade featured an arms-race towards faster and more robust keypoints and association algorithms: Scale Invariant Feature Transform (SIFT)[17], Speed-up Robust Feature (SURF)[4], and more recently Binary Robust Invariant Scalable Keypoints (BRISK)[I6] to name a few. These days, the deployment of vision algorithms on smart phones and embedded devices with low memory and computation complexity has even upped the ante: the goal is to make descriptors faster to compute, more compact while remaining robust to scale, rotation and noise. To best address the current requirements, we propose a novel keypoint descriptor inspired by the human visual system and more precisely the retina, coined Fast Retina Keypoint (FREAK). A cascade of binary strings is computed by efficiently comparing image intensities over a retinal sampling pattern. Our experiments show that FREAKs are in general faster to compute with lower memory load and also more robust than SIFT, SURF or BRISK. They are thus competitive alternatives to existing keypoints in particular for embedded applications.
Image compression with learnt tree-structured dictionaries

In the present paper, we propose a new framework for the construction of meaningful dictionaries for sparse representation of signals. The dictionary approach to coding and compression proves very attractive since decomposing a signal over a redundant set of basis functions allows a parsimonious representation of information. This interest is witnessed by numerous research efforts that have been done in the last years to develop an efficient algorithm for the decomposition of signals over redundant sets of functions. However, the effectiveness of such methods strongly depends on the dictionary and on its structure. In this work, we develop a method to learn overcomplete sets of functions from real-world signals. This technique allows the design of dictionaries that can be adapted to a specific class of signals. The found functions are stored in a tree structure. This data structure is used by a tree-based pursuit algorithm to generate sparse approximations of natural signals. Finally, the proposed method is considered in the context of image compression. Results show that the learning tree-based approach outperforms state-of-the-art coding technique.
A Bayesian approach to video expansions on parametric over-complete 2-D dictionaries

In this work, we explore a framework for the sparse representation of video sequences by means of spatio-temporal functions able to exploit the 2D nature of images and the temporal smoothness associated to object trajectories. Decomposition over redundant dictionaries formed by 2D functions capable to exploit image geometry, has shown to be well adapted for efficient sparse image approximations. Video representation by means of temporally evolving sets of such 2D functions seems thus a natural extension toward video approximation techniques. In the present paper we study the deformation of a geometry oriented image expansion based on matching pursuits (MP) (P. Vandergheynst and P. Frossard, May 2001), to obtain a parametric representation of frames transformation through time. We consider a modified MP approach based on a Bayesian decision criteria to deform geometrical primitives in a predictive fashion from frame to frame. Indeed, since motion stability is not guaranteed using a pure MP, a Bayesian framework is introduced to regularize motion among expansion terms of frames representations.
Directional dyadic wavelet transforms: design and algorithms

We propose a simple and efficient technique for designing translation invariant dyadic wavelet transforms (DWTs) in two dimensions. Our technique relies on an extension of the work of Duval-Destin et al. where dyadic decompositions are constructed starting from the continuous wavelet transform. The main advantage of this framework is that it allows for a lot of freedom in designing two-dimensional (2-D) dyadic wavelets. We use this property to construct directional wavelets, whose orientation filtering capabilities are very important in image processing. We address the efficient implementation of these decompositions by constructing approximate QMFs through an L(2) optimization. We also propose and study an efficient implementation in the Fourier domain for dealing with large filters.
On the exponential convergence of matching pursuits in quasi-incoherent dictionaries

The purpose of this correspondence is to extend results by Villemoes and Temlyakov about exponential convergence of Matching Pursuit (MP) with some structured dictionaries for "simple" functions in finite or infinite dimension. The results are based on an extension of Tropp's results about Orthogonal Matching Pursuit (OMP) in finite dimension, with the observation that it does not only work for OMP but also for MP. The main contribution is a detailed analysis of the approximation and stability properties of MP with quasi-incoherent dictionaries, and a bound on the number of steps sufficient to reach an error no larger than a penalization factor times the best m-term approximation error.
On the use of a priori information for sparse signal approximations

Recent results have underlined the importance of incoherence in redundant dictionaries for a good behavior of decomposition algorithms like matching and basis pursuit. However, appropriate dictionaries for a given application may not be able to meet the incoherence condition. In such a case, decomposition algorithms may completely fail in the retrieval of the sparsest approximation. This paper studies the effect of introducing a priori knowledge when recovering sparse approximations over redundant dictionaries. Theoretical results show how the use of reliable a priori information (which in this paper appears under the form of weights) can improve the performances of standard approaches such as greedy algorithms and relaxation methods. Our results reduce to the classical case when no prior information is available. Examples validate and illustrate our theoretical statements. EDICS: 2-NLSP
Sparse Image Reconstruction on the Sphere: Implications of a New Sampling Theorem

We study the impact of sampling theorems on the fidelity of sparse image reconstruction on the sphere. We discuss how a reduction in the number of samples required to represent all information content of a band-limited signal acts to improve the fidelity of sparse image reconstruction, through both the dimensionality and sparsity of signals. To demonstrate this result, we consider a simple inpainting problem on the sphere and consider images sparse in the magnitude of their gradient. We develop a framework for total variation inpainting on the sphere, including fast methods to render the inpainting problem computationally feasible at high resolution. Recently a new sampling theorem on the sphere was developed, reducing the required number of samples by a factor of two for equiangular sampling schemes. Through numerical simulations, we verify the enhanced fidelity of sparse image reconstruction due to the more efficient sampling of the sphere provided by the new sampling theorem.
Matching Pursuit With Block Incoherent Dictionaries

Recently, there has been an intense activity in the field of sparse approximations with redundant dictionaries, largely motivated by the practical performances of algorithms such as matching pursuit (MP) and basis pursuit (BP). However, most of the theoretical results obtained so far are valid only for the restricted class of incoherent dictionaries. This paper investigates a new class of overcomplete dictionaries, called block incoherent dictionaries, where coherence can be arbitrarily big. We show that a simple greedy algorithm can correctly identify stable subdictionaries (called blocks) and demonstrate how one can use the extra coherence freedom for approximation purposes.
Multiresolution segmentation of natural images: from linear to nonlinear scale-space representations

In this paper, we introduce a framework that merges classical ideas borrowed from scale-space and multiresolution segmentation with nonlinear partial differential equations. A nonlinear scale-space stack is constructed by means of an appropriate diffusion equation. This stack is analyzed and a tree of coherent segments is constructed based on relationships between different scale layers. Pruning this tree proves to be a very efficient tool for unsupervised segmentation of different classes of images (e.g., natural, medical, etc.). This technique is light on the computational point of view and can be extended to nonscalar data in a straightforward manner.
Graph-based denoising for time-varying point clouds

Noisy 3D point clouds arise in many applications. They may be due to errors when creating a 3D model from images or simply to imprecise depth sensors. Point clouds can be given geometrical structure using graphs created from the similarity information between points. This paper introduces a technique that uses this graph structure and convex optimization methods to denoise 3D point clouds. A short discussion presents how those methods naturally generalize to time-varying inputs such as 3D point cloud time series.
Adaptive Graph-Based Total Variation for Tomographic Reconstructions

Sparsity exploiting image reconstruction (SER) methods have been extensively used with total variation (TV) regularization for tomographic reconstructions. Local TV methods fail to preserve texture details and often create additional artifacts due to over-smoothing. Nonlocal TV (NLTV) methods have been proposed as a solution to this but they either lack continuous updates due to computational constraints or limit the locality to a small region. In this letter, we propose adaptive graph-based TV. The proposed method goes beyond spatial similarity between different regions of an image being reconstructed by establishing a connection between similar regions in the entire image regardless of spatial distance. As compared to NLTV, the proposed method is computationally efficient and involves updating the graph prior during every iteration making the connection between similar regions stronger. Moreover, it promotes sparsity in the wavelet and graph gradient domains. Since TV is a special case of graph TV, the proposed method can also be seen as a generalization of SER and TV methods.
Structured sparsity models for compressively sensed electrocardiogram signals: A comparative study

We have recently quantified and validated the potential of the emerging compressed sensing (CS) paradigm for real-time energy-efficient electrocardiogram (ECG) compression on resource-constrained sensors. In the present work, we investigate applying sparsity models to exploit underlying structural information in recovery algorithms. More specifically, re-visiting well-known sparse recovery algorithms, we propose novel model-based adaptations for the robust recovery of compressible signals like ECG. Our results show significant performance gains for the recovery algorithms exploiting the underlying sparsity models.
Towards stationary time-vertex signal processing

Graph-based methods for signal processing have shown promise for the analysis of data exhibiting irregular structure, such as those found in social, transportation, and sensor networks. Yet, though these systems are often dynamic, state-of-the-art methods for graph signal processing ignore the time dimension. To address this shortcoming, this paper considers the statistical analysis of time-varying graph signals. We introduce a novel definition of joint (time-vertex) stationarity, which generalizes the classical definition of time stationarity and the recent definition appropriate for graphs. This gives rise to a scalable Wiener optimization framework for denoising, semi-supervised learning, or more generally inverting a linear operator, that is provably optimal. Experimental results on real weather data demonstrate that taking into account graph and time dimensions jointly can yield significant accuracy improvements in the reconstruction effort.
Blind audiovisual separation based on redundant representations

In this work we present a method to perform a complete audiovisual source separation without need of previous information. This method is based on the assumption that sounds are caused by moving structures. Thus, an efficient representation of audio and video sequences allows to build relationships between synchronous structures on both modalities. A robust clustering algorithm groups video structures exhibiting strong correlations with the audio so that sources are counted and located in the image. Using such information and exploiting audio-video correlation, the audio sources activity is determined. Next, spectral Gaussian Mixture Models (GMMs) are learnt in time slots with only one source active so that it is possible to separate them in case of an audio mixture. Audio source separation performances are rigorously evaluated, clearly showing that the proposed algorithm performs efficiently and robustly.
Learning Multimodal Dictionaries

Real-world phenomena involve complex interactions between multiple signal modalities. As a consequence, humans are used to integrate at each instant perceptions from all their senses in order to enrich their understanding of the surrounding world. This paradigm can be also extremely useful in many signal processing and computer vision problems involving mutually related signals. The simultaneous processing of multimodal data can, in fact, reveal information that is otherwise hidden when considering the signals independently. However, in natural multimodal signals, the statistical dependencies between modalities are in general not obvious. Learning fundamental multimodal patterns could offer deep insight into the structure of such signals. In this paper, we present a novel model of multimodal signals based on their sparse decomposition over a dictionary of multimodal structures. An algorithm for iteratively learning multimodal generating functions that can be shifted at all positions in the signal is proposed, as well. The learning is defined in such a way that it can be accomplished by iteratively solving a generalized eigenvector problem, which makes the algorithm fast, flexible, and free of user-defined parameters. The proposed algorithm is applied to audiovisual sequences and it is able to discover underlying structures in the data. The detection of such audio-video patterns in audiovisual clips allows to effectively localize the sound source on the video in presence of substantial acoustic and visual distractors, outperforming state-of-the-art audiovisual localization algorithms.
Stationary Signal Processing on Graphs

Graphs are a central tool in machine learning and information processing as they allow to conveniently capture the structure of complex datasets. In this context, it is of high importance to develop flexible models of signals defined over graphs or networks. In this paper, we generalize the traditional concept of wide sense stationarity to signals defined over the vertices of arbitrary weighted undirected graphs. We show that stationarity is expressed through the graph localization operator reminiscent of translation. We prove that stationary graph signals are characterized by a well-defined power spectral density that can be efficiently estimated even for large graphs. We leverage this new concept to derive Wiener-type estimation procedures of noisy and partially observed signals and illustrate the performance of this new model for denoising and regression.
Fast Robust PCA on Graphs

Mining useful clusters from high dimensional data have received significant attention of the computer vision and pattern recognition community in the recent years. Linear and nonlinear dimensionality reduction has played an important role to overcome the curse of dimensionality. However, often such methods are accompanied with three different problems: high computational complexity (usually associated with the nuclear norm minimization), nonconvexity (for matrix factorization methods), and susceptibility to gross corruptions in the data. In this paper, we propose a principal component analysis (PCA) based solution that overcomes these three issues and approximates a low-rank recovery method for high dimensional datasets. We target the low-rank recovery by enforcing two types of graph smoothness assumptions, one on the data samples and the other on the features by designing a convex optimization problem. The resulting algorithm is fast, efficient, and scalable for huge datasets with O(n log(n)) computational complexity in the number of data samples. It is also robust to gross corruptions in the dataset as well as to the model parameters. Clustering experiments on 7 benchmark datasets with different types of corruptions and background separation experiments on 3 video datasets show that our proposed model outperforms 10 state-of-the-art dimensionality reduction models. Our theoretical analysis proves that the proposed model is able to recover approximate low-rank representations with a bounded error for clusterable data.
A locally temporal adaptive transform scheme for sub-band video coding

The presented work extends the concept of sub-band video coding based on a 3D wavelet transform to a more adaptive approach. A formal comparison is presented between the performances inferred by the use of the 3D wavelet transform and the use of a 2D wavelet in the spatial domain extended by a locally adaptive transform in the temporal dimension. Some advantages are foreseen for the new scheme since it is able to deal better with certain signal models, like appearing and moving edges. An increased control of distortion spreading is expected and, consequently, a lower visual impact relevance.
Hyperspectral image compressed sensing via low-rank and joint-sparse matrix recovery

We propose a novel approach to reconstruct Hyperspectral images from very few number of noisy compressive measurements. Our reconstruction approach is based on a convex minimization which penalizes both the nuclear norm and the ℓ2,1 mixed-norm of the data matrix. Thus, the solution tends to have a simultaneous low-rank and joint-sparse structure. We explain how these two assumptions fit Hyperspectral data, and by severals simulations we show that our proposed reconstruction scheme significantly enhances the state-of-the-art tradeoffs between the reconstruction error and the required number of CS measurements.
Light field compressive sensing in camera arrays

This paper presents a novel approach to capture light field in camera arrays based on the compressive sensing framework. Light fields are captured by a linear array of cameras with overlapping field of view. In this work, we design a redundant dictionary to exploit cross-cameras correlated structures to sparsely represent cameras image. Our main contributions are threefold. First, we exploit the correlations between the set of views by making use of a specially designed redundant dictionary. We show experimentally that the projection of complex scenes onto this dictionary yields very sparse coefficients. Second, we propose an efficient compressive encoding scheme based on the random convolution framework . Finally, we develop a joint sparse recovery algorithm for decoding the compressed measurements and show a marked improvement over independent decoding of CS measurements.
Blind Audiovisual Source Separation Based on Sparse Redundant Representations

In this paper, we propose a novel method which is able to detect and separate audiovisual sources present in a scene. Our method exploits the correlation between the video signal captured with a camera and a synchronously recorded one-microphone audio track. In a first stage, audio and video modalities are decomposed into relevant basic structures using redundant representations. Next, synchrony between relevant events in audio and video modalities is quantified. Based on this co-occurrence measure, audiovisual sources are counted and located in the image using a robust clustering algorithm that groups video structures exhibiting strong correlations with the audio. Next periods where each source is active alone are determined and used to build spectral Gaussian mixture models (GMMs) characterizing the sources acoustic behavior. Finally, these models are used to separate the audio signal in periods during which several sources are mixed. The proposed approach has been extensively tested on synthetic and natural sequences composed of speakers and music instruments. Results show that the proposed method is able to successfully detect, localize, separate, and reconstruct present audiovisual sources.
Fast TV-L1 optical flow for interactivity

Vision is a natural tool for human-computer interaction, since it provides visual feedback to the user and mimics some human behaviors. It requires however the fast and robust computation of motion primitives, which remains a difficult problem. In this work, we propose to apply some recent mathematical results about convex optimization to the TV-L1 optical flow problem. At the cost of a small smoothing of the Total Variation (TV), the convergence speed of the numerical scheme is improved, leading to earlier termination. Furthermore, we successfully implement our algorithm on GPU for realtime performance using the OpenCL framework. We demonstrate the potential of our optical flow by using it as primary sensor in a remotely controlled image browsing software.
Joint low-rank and sparse light field modelling for dense multiview data compression

The effective representation of the structures in the multiview images is an important problem that arises in visual sensor networks. This paper presents a novel recovery scheme from compressive samples which exploit local and non-local correlated structures in dense multiview images. The recovery model casts into convex minimization framework which penalizes the sparse and low-rank constraints on the data. The sparsity constraint models the correlations among pixels in a single image whereas the global correlations across images are modelled with the low-rank prior. Simulation results demonstrate that our approach achieves better reconstruct quality in comparison with the state-of-the-art reconstruction schemes.
Stream carving: An adaptive seam carving algorithm

We propose a new content-aware image resizing scheme, Stream Carving, which is based on the well-known seam carving method. Our algorithm may introduce larger seams in the retargeted image, i.e. seams with a width larger than one pixel, that we call “streams”. The resulting holes are then recovered using an inpainting method. Our retargeting algorithm is also more related to human perception by exploiting an adaptive importance map that merges several features like gradient magnitude, saliency, face, edge and straight line detection. Our approach induces an increase in the quality of the retargeted image when compared to the original seam carving method and provides similar or better results than other actual image retargeting techniques.