nils-herrmann/clean_scirepeval huggingface dataset:
Deformable Spectrograms

Speech and other natural sounds show high temporal correlation and smooth spectral evolution punctuated by a few, irregular and abrupt changes. In a conventional Hidden Markov Model (HMM), such structure is represented weakly and indirectly through transitions between explicit states representing ‘steps’ along such smooth changes. It would be more efficient and informative to model successive spectra astransformationsof their immediate predecessors, and we present a model which focuses on local deformations of adjacent bins in a timefrequency surface to explain an observed sound, using explicit representation only for those bins that cannot be predicted from their context. We further decompose the log-spectrum into two additive layers, which are able to separately explain and model the evolution of the harmonic excitation, and formant filtering of speech and similar sounds. Smooth deformations are modeled with hidden transformation variables in both layers, using Markov Random fields (MRFs) with overlapping subwindows as observations; inference is efficiently performed via loopy belief propagation. The model can fill-in deleted timefrequency cells without any signal model, and an entire signal can be compactly represented with a few specific states along with the deformation maps for both layers. We discuss several possible applications for this new model, including source separation.
Consistent segmentation for optical flow estimation

In this paper, we propose a method for jointly computing optical flow and segmenting video while accounting for mixed pixels (matting). Our method is based on statistical modeling of an image pair using constraints on appearance and motion. Segments are viewed as overlapping regions with fractional (alpha) contributions. Bidirectional motion is estimated based on spatial coherence and similarity of segment colors. Our model is extended to video by chaining the pairwise models to produce a joint probability distribution to be maximized. To make the problem more tractable, we factorize the posterior distribution and iteratively minimize its parts. We demonstrate our method on frame interpolation
Probabilistic Index Maps for Modeling Natural Signals

One of the major problems in modeling natural signals is that signals with very similar structure may locally have completely different measurements, e.g., images taken under different illumination conditions, or the speech signal captured in different environments. While there have been many successful attempts to address these problems in application-specific settings, we believe that underlying a large set of problems in signal representation is a representational deficiency of intensity-derived local measurements that are the basis of most efficient models. We argue that interesting structure in signals is better captured when the signal is defined as a matrix whose entries are discrete indices to a separate palette of possible measurements. In order to model the variability in signal structure, we define a signal class not by a single index map, but by a probability distribution over the index maps, which can be estimated from the data, and which we call probabilistic index maps. The existing algorithms can be adapted to work with this representation. Furthermore, the probabilistic index map representation leads to algorithms with computational costs proportional to either the size of the palette or the log of the size of the palette, making the cost of significantly increased invariance to non-structural changes quite bearable. We illustrate the benefits of the probabilistic index map representation in several applications in computer vision and speech processing.
Detailed graphical models for source separation and missing data interpolation in audio

Methods for blind source separation based only on general properties such as source independence encounter difficulties when the degree of overlap and/or the dimensionality of the observations make the blind inference problem unresolvable. Such situations require additional constraints on the form of the individual sources, motivating the development of models able to capture in detail the consistency and variability of a single source’s sound. With single-chain HMMs this requires a very large number of states, making the models marginally practical. Here we present two approaches to factorize the variability in detailed models. First is a coupled subband model, where each source signal is broken into multiple frequency bands, and separate but coupled HMMs are built for each band requiring many fewer states per model. In order to avoid the unnatural state combinations that would arise from independent models for each band (as in the multiband speech models of [1] and ), we couple adjacent bands resulting in a grid-like model (fig. 1) for the full spectrum. Exact inference of such a model is intractable, but we have derived an efficient approximation based on variational methods.
Hierarchical video clustering

We present a novel generative model for video that models video as mixture of transformed video scenes. The learning procedure automatically clusters video frames into video scenes and objects. The learning algorithm is based on a hierarchical, on-line EM algorithm. Fast Fourier transform (FFT) is used for rapid computations in E and M step of the EM algorithm. We use the model to: 1. perform video clustering by grouping similar (up to translation and scale) video frames into clusters; 2. robustly stabilize video by inferring translation and scale intensity for each frame. We believe that video scene modeling of this kind is essential to bridge the "semantic gap" in video understanding. We illustrate this with several excellent results, both in terms of speed and accuracy.
Probability models for high dynamic range imaging

Methods for expanding the dynamic range of digital photographs by combining images taken at different exposures have recently received a lot of attention. Current techniques assume that the photometric transfer function of a given camera is the same (modulo an overall exposure change) for all the input images. Unfortunately, this is rarely the case with today's camera, which may perform complex nonlinear color and intensity transforms on each picture. In this paper, we show how the use of probability models for the imaging system and weak prior models for the response functions enable us to estimate a different function for each image using only pixel intensity values. Our approach also allows us to characterize the uncertainty inherent in each pixel measurement. We can therefore produce statistically optimal estimates for the hidden variables in our model representing scene irradiance. We present results using this method to statistically characterize camera imaging functions and construct high-quality high dynamic range (HDR) images using only image pixel information.
Cryptographically secure identity certificates

We present FACECERTS, a simple, inexpensive, and cryptographically secure identity certification system. A FACECERT is a printout of person's portrait photo, an arbitrary textual message, and a 2D color bar-code which encodes an RSA signature of the message hash and the compressed representation of the face encompassed by the photo. The signature is created using the private key of the party issuing the ID. Verification is performed by a simple, intelligent, and off-line scanning device that contains the public key of the issuer. The system does not require smart cards. More interestingly, the ID does not need to be printed by a high-end printer, it can be printed anywhere. We present a novel algorithm for compressing faces and investigate the reliability of the crucial components of the system.
Audio-visual graphical models for speech processing

Perceiving sounds in a noisy environment is a challenging problem. Visual lip-reading can provide relevant information but is also challenging because lips are moving and a tracker must deal with a variety of conditions. Typically audio-visual systems have been assembled from individually engineered modules. We propose to fuse audio and video in a probabilistic generative model that implements cross-model self-supervised learning, enabling adaptation to audio-visual data. The video model features a Gaussian mixture model embedded in a linear subspace of a sprite which translates in the video. The system can learn to detect and enhance speech in noise given only a short (30 second) sequence of audio-visual data. We show some results for speech detection and enhancement, and discuss extensions to the model that are under investigation.
Capturing image structure with probabilistic index maps

One of the major problems in modeling images for vision tasks is that images with very similar structure may locally have completely different appearance, e.g., images taken under different illumination conditions, or the images of pedestrians with different clothing. While there have been many successful attempts to address these problems in application-specific settings, we believe that underlying a large set of problems in vision is a representational deficiency of intensity-derived local measurements that are the basis of most efficient models. We argue that interesting structure in images is better captured when the image is defined as a matrix whose entries are discrete indices to a separate palette of possible intensities, colors or other features, much like the image representation often used to save on storage. In order to model the variability in images, we define an image class not by a single index map, but by a probability distribution over the index maps, which can be automatically estimated from the data, and which we call probabilistic index maps. The existing algorithms can be adapted to work with this representation, as we illustrate in this paper on the example of transformation-invariant clustering and background subtraction. Furthermore, the probabilistic index map representation leads to algorithms with computational costs proportional to either the size of the palette or the log of the size of the palette, making the cost of significantly increased invariance to non-structural changes quite bearable.
Multiband audio modeling for single-channel acoustic source separation

Detailed hidden Markov models (HMMs) that capture the constraints implicit in a particular sound can be used to estimate obscured or corrupted portions from partial observations, the situation encountered when trying to identify multiple, overlapping sounds. However, when the complexity and variability of the sounds are high, as in a particular speaker's voice, a detailed model might require several thousand states to cover the full range of different short-term spectra with adequate resolution. To address the tractability problems of such large models, we break the source signals into multiple frequency bands, and build separate but coupled HMMs for each band, requiring many fewer states per model. To prevent non-natural full spectral states and to enforce consistency within and between bands, at any given frame, the state in a particular band is determined by the previous state in that band and the states in the adjacent bands. Coupling the bands in this manner results in a grid like model for the full spectrum. Since exact inference of such a model is intractable, we derive an efficient approximation based on variational methods. Results
Hierarchical transformation invariant clustering

We present a novel generative model for transformation invariant data clustering, with the emphasis on video clustering and summarization. We model the data as the mixture of transformed exemplars. We include in the model the most common and realistic transformations in video data: camera motion, cropping, change in scale and light. The learning procedure automatically clusters video frames into video scenes and objects. The learning algorithm is based on a hierarchical, on-line EM algorithm. Fast Fourier transform is used for rapid inference and learning. We use the model to: 1. perform video clustering by grouping similar (up to translation, scale and lighting) video frames into clusters; 2. robustly stabilize video frames by compensating for translation, scale and lighting changes. We believe that video scene modelling of this kind is essential to bridge the semantic gap in video understanding. We illustrate this with several excellent results, both in terms of speed and accuracy.
Towards single-channel unsupervised source separation of speech mixtures: the layered harmonics/formants separation-tracking model

Speaker models for blind source separation are typically based on HMMs consisting of vast numbers of states to capture source spectral variation, and trained on large amounts of isolated speech. Since observations can be similar between sources, inference relies on sequential constraints from the state transition matrix which are, however, quite weak. To avoid these problems, we propose a strategy of capturing local deformations of the time-frequency energy distribution. Since consecutive spectral frames are highly correlated, each frame can be accurately described as a nonuniform deformation of its predecessor. A smooth pattern of deformations is indicative of a single speaker, and the cliffs in the deformation fields may indicate a speaker switch. Further, the log-spectrum of speech can be decomposed into two additive layers, separately describing the harmonics and formant structure. We model smooth deformations as hidden transformation variables in both layers, using MRFs with overlapping subwindows as observations, assumed to be a noisy sum of the two layers. Loopy belief propagation provides for efficient inference. Without any pre-trained speech or speaker models, this approach can be used to fill in missing time-frequency observations, and the local entropy of the deformation fields indicate source boundaries for separation.
Tamper-Resistant Biometric IDs

We present FaceCerts, a simple, inexpensive, and cryptographically secure identity certification system. A FaceCert is a printout of person’s portrait photo, an arbitrary textual message, and a 2-D color barcode which encodes an RSA signature of the message hash and the compressed representation of the face encompassed by the photo. The signature is created using the private key of the party issuing the ID. ID verification is performed by a simple off-line scanning device that contains the public key of the issuer. The system does not require smart cards; it can be expanded to encompass other biometric features, and more interestingly, the ID does not need to be printed by a trusted or high-end printer, it can be printed anywhere, anytime, and potentially by anyone. The ID verifier uses a single scan process which does not require the use of displays. We detail system’s components and present a preliminary performance evaluation using an in-field experiment.
Scene generative models for adaptive video fast forward

In this paper, we present a statistical generative model of scenes with multiple objects that can be efficiently used for tasks related to video search, browsing and retrieval. Instead of using a combination of weighted Euclidean distances as a shot similarity measure, we base the retrieval process on the likelihood of a video frame under the generative model trained on the query sequence. This allows for automatic separation and balancing of various causes of variability, such as occlusion, appearance change and motion. In previous work, this usually required complex user intervention. The likelihood models we study in this paper are based on appearances of multiple, possibly occluding objects in a video clip. Given a query, the video is played at a higher rate until similar frames are found. The playback speed is linked to the frame likelihood, so that the speed drops as the likely target frames are starting to come in.
Ieee Transactions on Signal Processing: Supplement on Secure Media 1 Facecerts Ieee Transactions on Signal Processing: Supplement on Secure Media 2

We present FACECERTS, a simple, inexpensive, and cryptographically secure identity certification system. A FACECERT is a printout of person’s portrait photo, an arbitrary textual message, and a 2D color bar-code which encodes a cryptographic signature of the message hash and the compressed representation of the face encompassed by the photo. The signature is created using the private key of the party issuing the ID. Verification is performed by a simple, intelligent, and off-line scanning device that contains the public key of the issuer. The system does not require smart cards. More interestingly, the ID does not need to be printed by a trusted or high-end printer, it can be printed anywhere, anytime, and potentially by anyone. In this paper, we present the crucial component of the FACECERT system, a novel hybrid algorithm for compressing faces, and investigate system reliability and performance.
A generative model for 2 . 5 D vision : Estimating appearance , transformation , illumination , transparency and occlusion

Most vision research is based on some sort of a model of the world. The model is usually highly specific about one aspect of the world, for instance, the appearance of a human face, or the motion model for an object, but it does not model other parts of the scene. This makes models very brittle in any realistic application. We believe that in order to build more robust vision algorithms, models need to be capable of capturing different aspects of the data at the same time, be fairly simple, but adapt to the data. Flexible models, as defined in the machine learning community, are minimally structured probability models with a large number of parameters that can adapt so as to explain the input data. We describe a framework for designing and using flexible models for vision. The framework uses structured probability models to describe causes of variability in the data, variational methods for probabilistic inference and generalized expectation-maximization algorithm for parameter estimation. We show that within this framework, we can perform various vision tasks jointly, such as tracking, recognition, occlusion detection, object stabilization, object removal, filtering, illumination stabilization, panoramic image/video computation, artistic rendering, etc. In fact, we argue that dealing with these tasks jointly is easier than combining individually optimized modules in a typical engineering approach to signal processing.
Detection and estimation of pointing gestures in dense disparity maps

We describe a real-time system for detecting pointing gestures and estimating the direction of pointing using stereo cameras. Previously, similar systems were implemented using color-based blob trackers, which relied on effective skin color detection; this approach is sensitive to lighting changes and the clothing worn by the user. In contrast, we used a stereo system that produces dense disparity maps in real-time. Disparity maps are considerably less sensitive to lighting changes. Our system subtracts the background, analyzes the foreground pixels to break the body into parts using a robust mixture model, and estimates the direction of pointing. We have tested the system on both coarse and fine pointing by selecting the targets in a room and controlling the cursor on a wall screen, respectively.
Computer Vision and Graphics Techniques for Modeling Dressed Humans

In this chapter we present techniques for building dressed human models from images. We combine computer vision based approaches such as 3-D reconstruction of a human body and analysis-by-synthesis of the behavior of cloth material with the computer graphics approaches for realistic rendering of complex objects. The experimental results include building textured 3-D models of humans from multiple images, dressing these models into virtual garment, and joint estimation of cloth draping parameters and the underlying object’s geometry in range images of dressed objects.
A Framework for Garment Shopping over the Internet

In this chapter, we propose a framework for integrated design, advertisement and retailing of garments over the Internet. This on-line shopping framework would make use of the latest research in computer graphics, image processing, computer vision and artificial intelligence. We describe these technologies in more detail, and explain how they can be used to build a visually attractive and easy to use interface to an intelligent integrated system that fulfils most of the functions of the traditional production chain, while allowing for mass-customization.
Tracking self-occluding articulated objects in dense disparity maps

In this paper, we present an algorithm for real-time tracking of articulated structures in dense disparity maps derived from stereo image sequences. A statistical image formation model that accounts for occlusions plays the central role in our tracking approach. This graphical model (a Bayesian network) assumes that the range image of each part of the structure is formed by drawing the depth candidates from a 3-D Gaussian distribution. The advantage over the classical mixture of Gaussians is that our model takes into account occlusions by picking the minimum depth (which could be regarded as a probabilistic version of z-buffering). The model also enforces articulation constraints among the parts of the structure. The tracking problem is formulated as an inference problem in the image formation model. This model can be extended and used for other tasks in addition to the one described in the paper and can also be used for estimating probability distribution functions instead of the ML estimates of the tracked parameters. For the purposes of real-time tracking, we used certain approximations in the inference process, which resulted in a real-time two-stage inference algorithm. We were able to successfully track upper human body motion in real time and in the presence of self-occlusions.
Tracking Articulated Structures in Stereo Image Sequences

| In this paper, we present an algorithm for real time 3-D tracking of articulated structures in stereo image sequences. These sequences can be captured by an inexpensive commercially available system that also computes the dense disparity map in real time. In our algorithm, the tracked object is modeled as a set of articulated 3D blobs, each adhering to a Gaussian distribution. Classi cation of the disparity map pixels into the segments of the articulated object is based on the maximum likelihood principle with an additional mechanism for lling the missing data created by self-occlusions. The articulation constraints are enforced through an Extended Kalman Filter, which can also be used to model the dynamics of the tracked object.
Computer modeling, analysis, and synthesis of dressed humans

We present computer vision techniques for building dressed human models using images. We develop an algorithm for three-dimensional body reconstruction and texture mapping using contour, stereo, and texture information from several images and deformable superquadrics as the model parts. We demonstrate a novel vision technique for analysis of cloth draping behavior. This technique allows for estimation of cloth model parameters, such as bending properties, but can also be used to estimate the contact points between the body and clothing in the range data of dressed humans. Combined with our body reconstruction algorithm and additional constraints on the articulation model, the detection of the garment-body contact points allows construction of a dressed human model in which even the geometry that was covered by clothing in the available data is reasonably well estimated.