nils-herrmann/clean_scirepeval huggingface dataset:
Towards computational models of kinship verification

We tackle the challenge of kinship verification using novel feature extraction and selection methods, automatically classifying pairs of face images as “related” or “unrelated” (in terms of kinship). First, we conducted a controlled online search to collect frontal face images of 150 pairs of public figures and celebrities, along with images of their parents or children. Next, we propose and evaluate a set of low-level image features for this classification problem. After selecting the most discriminative inherited facial features, we demonstrate a classification accuracy of 70.67% on a test set of image pairs using K-Nearest-Neighbors. Finally, we present an evaluation of human performance on this problem.
Audio visual interaction in multimedia

Audio-visual interaction is a very important issue in personal communication applications. The research and development of multimedia communication systems should account for this interaction. In this paper, we address a number of areas related to audio-visual interaction, such as automatic lipreading, speech-driven talking heads, and lip synchronization. In particular, we will discuss a new trend in video coding research: joint audio-video coding. Given that mouth movements are very difficult to code because of its rapid, complex, and non-rigid motion (so conventional block-based motion-compensation methods fail), we will explain how having extra help from the acoustic signal can enable us to code the mouth movements more efficiently.
An investigation into subspace rapid speaker adaptation for verification

Rapid speaker adaptation is becoming more important in emerging applications where storage, computation and training utterances are at a premium (e.g. PDAs, cell phones). Effective adaptation can be achieved for the task of speaker verification, based on a maximum a posteriori (MAP) learning framework, by restricting the client's parametric model to be a linear combination of parameters estimated from training observations and a speaker independent "world" model (i.e. relevance adaptation (RA)). Subspace adaptation (SA) attempts to restrict a client's parametric representation to a pre-defined subspace during estimation. In this paper we elucidate where subspace adaptation outperforms world adaptation, demonstrate where and why subspace adaptation is sometimes not as effective and give insights into what cost criteria should be used to construct the adaptation parametric subspace. Results are presented on the acoustic portion of the XM2VTS database for the task of Gaussian mixture model (GMM) based text-independent speaker verification.
Improving subpixel stereo matching with segment evolution

Segmentation-based approach has shown significant success in stereo matching. By assuming pixels within one image segment belong to the same 3D surface, robust depth estimation can be achieved by taking the whole segment into consideration. However, segmentation has been mostly used for stereo matching at integer disparities rather than subpixel disparities. One major reason is that small segments may be insufficient for estimating surfaces like slanted planes, while large segments may contain segmentation errors impacting the accuracy of depth estimation. In this work, we propose a segmentation-based scheme for subpixel stereo matching. Instead of using a fixed segmentation, segments are evolved to find a better support for reliable surface estimation. Given an initial estimation of segmentation and depth, the proposed algorithm jointly optimizes the segmentation and depth by evolving the segmentation at the pixel level and updating the plane parameters at the segment level. Justified with experiments performed on the Middlebury benchmark, we show that the proposed method achieves significant improvements for subpixel stereo matching.
Shaping for video with frame dependency

Streaming of preceded video, which is both source- and channel-coded, over packet-loss networks faces challenges of the time-varying packet loss rate and fluctuating bandwidth. Rate shaping has been proposed to reduce the bit rate of a preceded video bitstream to adapt to the real-time bandwidth variation. In our earlier work, rate shaping was extended to consider not only the bandwidth but also the packet loss rate variations. In practice, the reconstructed result of the previous frame will affect the following frames if the video is predictive coded, and/or the error concealment method performed at the receiver utilizes temporal information. However, none prior work considers rate shaping for pre source- and channel-coded video with such frame dependency. In order to incorporate frame dependency into the rate shaping process, we propose to send the location (and mean) of the corrupted macroblock back to the sender, and use such feedback information to determine the distortion measure in the rate-distortion optimized rate shaping. Experiments have shown improved performance with the aids of the feedback information.
Realistic video avatar

We present a system for the implementation of a photorealistic avatar using video captured from a user. This is achieved by constructing the dynamic video texture map and combining it with the 3D mesh model of the user to render the photorealistic avatar. The dynamic video texture map reflects the user's facial expressions and is generated by composing the cylindrical texture map of the user with constant updates either directly from the input video or derived from a temporal and spatial interpolation scheme. To derive the 3D mesh model for the user, we define feature points on a generic model and use the feature points identified on the video sequence to deform and obtain the avatar model. The goal of this realistic video avatar project is to provide a vivid representation of participants with a more realistic quality avatar, as compared to using only the facial animation parameters defined in MPEG-4 without the corresponding image updates. This scheme is suitable for conferencing applications because it requires much lower bandwidth than live video, and yet provides a 3D avatar representation for any virtual environment.
Seed Image Selection in interactive cosegmentation

Interactive image segmentation is a powerful paradigm that allows users to direct the segmentation algorithm towards a desired output. However, marking scribbles on multiple images is a cumbersome process. Recent works show that statistics collected from user input in a single image can be shared among a group of related images to perform interactive cosegmentation. Most works use a naive heuristic of requesting the user input on a random image from the group. We show that in practice, selecting the right image to scribble on is critical to the resulting segmentation quality. In this paper, we address the problem of Seed Image Selection, i.e., deciding which image among a group of related images should be presented to the user for scribbling. We formulate our approach as a classification problem and show that our approach outperforms the naive heuristic used by other works.
Towards Understanding User Preferences from User Tagging Behavior for Personalization

Personalizing image tags is a relatively new and growing area of research, and in order to advance this research community, we must review and challenge the de-facto standard of defining tag importance. We believe that for greater progress to be made, we must go beyond tags that merely describe objects that are visually represented in the image, towards more user-centric and subjective notions such as emotion, sentiment, and preferences. We focus on the notion of user preferences and show that the order that users list tags on images is correlated to the order of preference over the tags that they provided for the image. While this observation is not completely surprising, to our knowledge, we are the first to explore this aspect of user tagging behavior systematically and report empirical results to support this observation. We argue that this observation can be exploited to help advance the image tagging (and related) communities. Our contributions include: 1.) conducting a user study demonstrating this observation, 2.) collecting a dataset with user tag preferences explicitly collected.
Soft shape context for iterative closest point registration

This paper introduces a shape descriptor, the soft shape context, motivated by the shape context method. Unlike the original shape context method, where each image point was hard assigned into a single histogram bin, we instead allow each image point to contribute to multiple bins, hence more robust to distortions. The soft shape context can easily be integrated into the iterative closest point (ICP) method as an auxiliary feature vector, enriching the representation of an image point from spatial information only, to spatial and shape information. This yields a registration method more robust than the original ICP method. The method is general for 2D shapes. It does not calculate derivatives, hence being able to handle shapes with junctions and discontinuities. We present experimental results to demonstrate the robustness compared with the standard ICP method.
Toward correlating and solving abstract tasks using convolutional neural networks

Most works using convolutional neural networks (CNN) show the efficacy of their methods in standard object recognition tasks, but not in abstract tasks such as emotion classification and memorability prediction, which are a subject of increasing importance (especially as machines become more autonomous, there is a need for semantic understanding). To verify whether CNN-based methods are effective in abstract tasks, we select 8 different abstract tasks in computer vision, evaluating the performance of 5 different CNN-based training approaches in these tasks. We show that CNN-based approaches outperform the state-of-the-art results in all the 8 tasks. Furthermore, we show that concatenating CNN features learned from different tasks can enhance the performance in each task. We also show that concatenating the CNN features learned from all the tasks under experiment does not perform the best, which is different from what is usually shown in previous works. Using CNN as a tool to correlate different tasks, we suggest which CNN features researchers should use in each task.
A new frame interpolation scheme for talking head sequences

A video codec typically-skips frames to satisfy the bit rate constraint. This results in a jerky motion and a loss of lip synchronization in sequences of talking persons. We propose a frame interpolation technique to solve these problems. The main components of this technique include: foreground/background segmentation, mesh-based frame interpolation, and image analysis/synthesis for mouth movements. This technique generates smooth head motion and renders lip motion that is synchronized with the voice of the person.
On the choice of rational decimation systems for multidimensional signals

Rational decimation systems, which alter the signal sampling rates by a rational factor, can be used to reduce the data rate of bandlimited signals without loss of information. In the present work, this idea is extended from the one-dimensional case to the multidimensional (MD) case. Given an arbitrary support of bandlimited MD signals, it is shown how to find a rational matrix whose irreducible left matrix fraction description defines the decimation and expansion matrices required in the rational decimation system. It is also shown how to choose the filter in this system to completely eliminate image components and aliasing effects. This technique is also applied to MD signals that are bandlimited in hyperspherical supports.<<ETX>>
A Hierarchical Visual Model for Video Object Summarization

We propose a novel method for removing irrelevant frames from a video given user-provided frame-level labeling for a very small number of frames. We first hypothesize a number of windows which possibly contain the object of interest, and then determine which window(s) truly contain the object of interest. Our method enjoys several favorable properties. First, compared to approaches where a single descriptor is used to describe a whole frame, each window's feature descriptor has the chance of genuinely describing the object of interest; hence it is less affected by background clutter. Second, by considering the temporal continuity of a video instead of treating frames as independent, we can hypothesize the location of the windows more accurately. Third, by infusing prior knowledge into the patch-level model, we can precisely follow the trajectory of the object of interest. This allows us to largely reduce the number of windows and hence reduce the chance of overfitting the data during learning. We demonstrate the effectiveness of the method by comparing it to several other semi-supervised learning approaches on challenging video clips.
On modeling variations for face authentication

We present a scheme for face authentication in the presence of variations. To deal with variations, such as facial expressions and registration errors, with which traditional appearance-based methods do not perform well, we propose the eigenflow approach. In this approach, the optical flow and the optical flow residue between a test image and a training image are computed first. The optical flow is then fitted to a model that is pre-trained by applying principal component analysis (PCA) to optical flows resulting from variations caused by facial expressions and registration errors. The eigenflow residue, optimally combined with the optical flow residue using linear discriminant analysis (LDA), determines the authenticity of the test image. Experimental results show that the proposed scheme outperforms the traditional methods in the presence of expression variations and registration errors. The approach can be extended to model lighting and pose variations as well.
Model-based error concealment for wireless video

We introduce model-based schemes for error concealment of networked video. We build appearance models for specific objects in the scene and use these models to replenish any lost information. Due to the models being designed specifically for the object, they are able to capture the statistical variations in the object appearance more effectively, thereby leading to better error-concealment performance. We examine statistical modeling techniques from the literature and introduce a new efficient and accurate linear model for data representation, called the mixture of principal components (MPC), and use these models for error concealment. We simulate lossy network conditions and show that these model-based concealment schemes outperform traditional concealment schemes across a variety of loss probabilities and bit rates for the coded video.
From appearance to context-based recognition: Dense labeling in small images

Traditionally, object recognition is performed based solely on the appearance of the object. However, relevant information also exists in the scene surrounding the object. As supported by our human studies, this contextual information is necessary for accurate recognition in low resolution images. This scenario with impoverished appearance information, as opposed to using images of higher resolution, provides an appropriate venue for studying the role of context in recognition. In this paper, we explore the role of context for dense scene labeling in small images. Given a segmentation of an image, our algorithm assigns each segment to an object category based on the segmentpsilas appearance and contextual information. We explicitly model context between object categories through the use of relative location and relative scale, in addition to co-occurrence. We perform recognition tests on low and high resolution images, which vary significantly in the amount of appearance information present, using just the object appearance information, the combination of appearance and context, as well as just context without object appearance information (blind recognition). We also perform these tests in human studies and analyze our findings to reveal interesting patterns. With the use of our context model, our algorithm achieves state-of-the-art performance on MSRC and Corel. datasets.
New video and multimedia standards and their impact to implementation

In the development of video and multimedia standards, new algorithms often result from the need to improve the efficiency or to reduce the cost of implementation. On the other hand, new implementation techniques are often triggered by algorithms that are too computation intensive with existing technology. Therefore, in multimedia applications, the development of algorithms and the progress in architectures are closely tied to each other. In this paper, we will introduce a number of new video and multimedia standards and discuss special considerations demanded by these new standards.
Cross-modal prediction in audio-visual communication

We present a novel means for predicting the shape of a person's mouth from the corresponding speech signal and explore applications of this prediction to video coding. The prediction is accomplished by modeling the probability distribution of the audiovisual features by a Gaussian mixture density. The optimal estimate for the visual features given the acoustic features can then be computed using this probability distribution. The ability to predict a person's mouth shape from the corresponding audio leads to a number of interesting joint audio-video coding strategies. In the cross-modal predictive coding system described, a model-based video coder compares measured visual parameters with predicted visual parameters, and sends the difference between the two to the receiver. Since the decoder also receives the acoustic data, it can form the prediction and then reconstruct the original parameters by adding the transmitted error signal.
Efficient inference for fully-connected CRFs with stationarity

The Conditional Random Field (CRF) is a popular tool for object-based image segmentation. CRFs used in practice typically have edges only between adjacent image pixels. To represent object relationship statistics beyond adjacent pixels, prior work either represents only weak spatial information using the segmented regions, or encodes only global object co-occurrences. In this paper, we propose a unified model that augments the pixel-wise CRFs to capture object spatial relationships. To this end, we use a fully connected CRF, which has an edge for each pair of pixels. The edge potentials are defined to capture the spatial information and preserve the object boundaries at the same time. Traditional inference methods, such as belief propagation and graph cuts, are impractical in such a case where billions of edges are defined. Under only one assumption that the spatial relationships among different objects only depend on their relative positions (spatially stationary), we develop an efficient inference algorithm that converges in a few seconds on a standard resolution image, where belief propagation takes more than one hour for a single iteration.
A GMM parts based face representation for improved verification through relevance adaptation

Motivated by the success of parts based representations in face detection we have attempted to address some of the problems associated with applying such a philosophy to the task of face verification. Hitherto, a major problem with this approach in face verification is the intrinsic lack of training observations, stemming from individual subjects, in order to estimate the required conditional distributions. The estimated distributions have to be generalized enough to encompass the differing permutations of a subject's face yet still be able to discriminate between subjects. In our work the well known Gaussian mixture model (GMM) framework is employed to model the conditional density function of the parts based representation of the face. We demonstrate that excellent performance can be obtained from our GMM based representation through the employment of adaptation theory, specifically relevance adaptation (RA). Our results are presented for the frontal images of the BANCA database.
Robust Parking Space Detection Considering Inter-Space Correlation

A major problem in metropolitan areas is searching for parking spaces. In this paper, we propose a novel method for parking space detection. Given input video captured by a camera, we can distinguish the empty spaces from the occupied spaces by using an 8-class support vector machine (SVM) classifier with probabilistic outputs. Considering the inter-space correlation, the outputs of the SVM classifier are fused together using a Markov random field (MRF) framework. The result is much improved detection performance, even when there are significant occlusion and shadowing effects in the scene. Experimental results are given to show the robustness of the proposed approach.
Camera-based clear path detection

In using image analysis to assist a driver to avoid obstacles on the road, traditional approaches rely on various detectors designed to detect different types of objects. We propose a framework that is different from traditional approaches in that it focuses on finding a clear path ahead. We assume that the video camera is calibrated offline (with known intrinsic and extrinsic parameters) and vehicle information (vehicle speed and yaw angle) is known. We first generate perspective patches for feature extraction in the image. Then, after extracting and selecting features of each patch, we estimate an initial probability that the patch corresponds to clear path using a support vector machine (SVM) based probability estimator on the selected features. We finally perform probabilistic patch smoothing based on spatial and temporal constraints to improve the initial estimate, thereby enhancing detection performance. We show that the proposed framework performs well even in some challenging examples with shadows and illumination changes.
A latent social approach to YouTube popularity prediction

Current works on Information Centric Networking assume the spectrum of caching strategies under the Least Recently/Frequently Used (LRFU) scheme as the de-facto standard, due to the ease of implementation and easier analysis of such strategies. In this paper we predict the popularity distribution of YouTube videos within a campus network. We explore two broad approaches in predicting the popularity of videos in the network: consensus approaches based on aggregate behavior in the network, and social approaches based on the information diffusion over an implicit network. We measure the performance of our approaches under a simple caching framework by picking the k most popular videos according to our predicted distribution and calculating the hit rate on the cache. We develop our approach by first incorporating video inter-arrival time (based on the power-law distribution governing the transmission time between two receivers of the same message in scale-free networks) to the baseline (LRFU), then combining with an information diffusion model over the inferred latent social graph that governs diffusion of videos in the network. We apply techniques from latent social network inference to learn the sharing probabilities between users in the network and apply a virus propagation model borrowed from mathematical epidemiology to estimate the number of times a video will be accessed in the future. Our approach gives rise to a 14% hit rate improvement over the baseline.