nils-herrmann/clean_scirepeval huggingface dataset:
How everyday visual experience prepares the way for learning object names

Infants learn their first object names by linking heard names to scenes. A core theoretical problem is how infants select the right referent from cluttered and ambiguous scenes. Here we show how the distributional properties of objects in young infants' visual experiences may help solve this core problem in early word learning. Infant perspective scenes of mealtimes were collected using head cameras worn by 9-month-old infants (147 mealtimes from 8 infants). The frequency distribution of objects was extremely skewed with the most frequent visual objects corresponding to the normatively first learned object names in English.
Fast Asymmetric Learning for Cascade Face Detection

A cascade face detector uses a sequence of node classifiers to distinguish faces from nonfaces. This paper presents a new approach to design node classifiers in the cascade detector. Previous methods used machine learning algorithms that simultaneously select features and form ensemble classifiers. We argue that if these two parts are decoupled, we have the freedom to design a classifier that explicitly addresses the difficulties caused by the asymmetric learning goal. There are three contributions in this paper: The first is a categorization of asymmetries in the learning goal and why they make face detection hard. The second is the forward feature selection (FFS) algorithm and a fast precomputing strategy for AdaBoost. FFS and the fast AdaBoost can reduce the training time by approximately 100 and 50 times, in comparison to a naive implementation of the AdaBoost feature selection method. The last contribution is a linear asymmetric classifier (LAC), a classifier that explicitly handles the asymmetric learning goal as a well-defined constrained optimization problem. We demonstrated experimentally that LAC results in an improved ensemble classifier performance.
Model-based tracking of self-occluding articulated objects

Computer sensing of hand and limb motion is an important problem for applications in human computer interaction and computer graphics. We describe a framework for local trading of self occluding motion, in which one part of an object obstructs the visibility of another. Our approach uses a kinematic model to predict occlusions and windowed templates to track partially occluded objects. We present offline 3D tracking results for hand motion with significant self occlusion.<<ETX>>
Quasi-periodic event analysis for social game retrieval

A new problem of retrieving social games from unstructured videos is proposed. Social games are characterized by repetitions (with variations) of alternating turns between two players. We define games as quasi-periodic motion patterns in video based on their repetitiveness property. We have developed an algorithm to extract such patterns from video. The patterns extracted by our method, from video clips of social games taken from YouTube, are shown to correspond to meaningful stages of the games. We demonstrate promising results in retrieving social games from unstructured, lab-recorded footage of children's play, and identifying social interactions in a dataset of approximately 3.75 hours of home movies.
Shadow elimination and occluder light suppression for multi-projector displays

Two related problems of front projection displays, which occur when users obscure a projector, are: (i) undesirable shadows cast on the display by the users, and (ii) projected light falling on and distracting the users. This paper provides a computational framework for solving these two problems based on multiple overlapping projectors and cameras. The overlapping projectors are automatically aligned to display the same dekeystoned image. The system detects when and where shadows are cast by occluders and is able to determine the pixels, which are occluded in different projectors. Through a feedback control loop, the contributions of unoccluded pixels from other projectors are boosted in the shadowed regions, thereby eliminating the shadows. In addition, pixels, which are being occluded, are blanked, thereby preventing the projected light from falling on a user when they occlude the display. This can be accomplished even when the occluders are not visible to the camera. The paper presents results from a number of experiments demonstrating that the system converges rapidly with low steady-state errors.
${\rm C}^{4}$: A Real-Time Object Detection Framework

A real-time and accurate object detection framework, C4, is proposed in this paper. C4 achieves 20 fps speed and the state-of-the-art detection accuracy, using only one processing thread without resorting to special hardware such as GPU. The real-time accurate object detection is made possible by two contributions. First, we conjecture (with supporting experiments) that contour is what we should capture and signs of comparisons among neighboring pixels are the key information to capture contour cues. Second, we show that the CENTRIST visual descriptor is suitable for contour based object detection, because it encodes the sign information and can implicitly represent the global contour. When CENTRIST and linear classifier are used, we propose a computational method that does not need to explicitly generate feature vectors. It involves no image preprocessing or feature vector normalization, and only requires O(1) steps to test an image patch. C4 is also friendly to further hardware acceleration. It has been applied to detect objects such as pedestrians, faces, and cars on benchmark data sets. It has comparable detection accuracy with state-of-the-art methods, and has a clear advantage in detection speed.
A Scalable Approach to Activity Recognition based on Object Use

We propose an approach to activity recognition based on detecting and analyzing the sequence of objects that are being manipulated by the user. In domains such as cooking, where many activities involve similar actions, object-use information can be a valuable cue. In order for this approach to scale to many activities and objects, however, it is necessary to minimize the amount of human-labeled data that is required for modeling. We describe a method for automatically acquiring object models from video without any explicit human supervision. Our approach leverages sparse and noisy readings from RFID tagged objects, along with common-sense knowledge about which objects are likely to be used during a given activity, to bootstrap the learning process. We present a dynamic Bayesian network model which combines RFID and video data to jointly infer the most likely activity and object labels. We demonstrate that our approach can achieve activity recognition rates of more than 80% on a real-world dataset consisting of 16 household activities involving 33 objects with significant background clutter. We show that the combination of visual object recognition with RFID data is significantly more effective than the RFID sensor alone. Our work demonstrates that it is possible to automatically learn object models from video of household activities and employ these models for activity recognition, without requiring any explicit human labeling.
Social interactions: A first-person perspective

This paper presents a method for the detection and recognition of social interactions in a day-long first-person video of u social event, like a trip to an amusement park. The location and orientation of faces are estimated and used to compute the line of sight for each face. The context provided by all the faces in a frame is used to convert the lines of sight into locations in space to which individuals attend. Further, individuals are assigned roles based on their patterns of attention. The rotes and locations of individuals are analyzed over time to detect and recognize the types of social interactions. In addition to patterns of face locations and attention, the head movements of the first-person can provide additional useful cues as to their attentional focus. We demonstrate encouraging results on detection and recognition of social interactions in first-person videos captured from multiple days of experience in amusement parks.
Exemplar-based segmentation of pigmented skin lesions from dermoscopy images

Automated segmentation of pigmented skin lesions (PSLs) from dermoscopy images is an important step for computer-aided diagnosis of skin cancer. The segmentation task involves classifying each image pixel as either lesion or skin. It is challenging because lesion and skin can often have similar appearance. We present a novel exemplar-based algorithm for lesion segmentation which leverages the context provided by a global color model to retrieve annotated examples which are most similar to a given query image. Pixel labels are generated through a probabilistic voting rule and smoothed using a dermoscopy-specific spatial prior. We compare our method to three competing techniques using a large dataset of dermoscopy images with hand-segmented ground truth,We show that our exemplar-based approach yields significantly better segmentations and is computationally efficient.
Visual 3D tracking of child-adult social interactions

We describe an approach to continuously capture children's 3D head pose and location during a tabletop social interaction with an adult examiner. Our approach, called face plus context, utilizes a fixed room camera in conjunction with a head-worn camera on the examiner to simultaneously capture the child's face along with the toys and social partners that provide context. Our system performs head tracking and pose estimation along with multi-target tracking to provide 3D localization and disambiguate identity. We evaluated our method on a dataset of 16 children, including both typically developing and autistic children. We present encouraging results for measuring children's social behaviors, along with validation results using an IMU.
Unsupervised Learning of Edges

Data-driven approaches for edge detection have proven effective and achieve top results on modern benchmarks. However, all current data-driven edge detectors require manual supervision for training in the form of hand-labeled region segments or object boundaries. Specifically, human annotators mark semantically meaningful edges which are subsequently used for training. Is this form of strong, highlevel supervision actually necessary to learn to accurately detect edges? In this work we present a simple yet effective approach for training edge detectors without human supervision. To this end we utilize motion, and more specifically, the only input to our method is noisy semi-dense matches between frames. We begin with only a rudimentary knowledge of edges (in the form of image gradients), and alternate between improving motion estimation and edge detection in turn. Using a large corpus of video data, we show that edge detectors trained using our unsupervised scheme approach the performance of the same methods trained with full supervision (within 3-5%). Finally, we show that when using a deep network for the edge detector, our approach provides a novel pre-training scheme for object detection.
Learning to recognize objects in egocentric activities

This paper addresses the problem of learning object models from egocentric video of household activities, using extremely weak supervision. For each activity sequence, we know only the names of the objects which are present within it, and have no other knowledge regarding the appearance or location of objects. The key to our approach is a robust, unsupervised bottom up segmentation method, which exploits the structure of the egocentric domain to partition each frame into hand, object, and background categories. By using Multiple Instance Learning to match object instances across sequences, we discover and localize object occurrences. Object representations are refined through transduction and object-level classifiers are trained. We demonstrate encouraging results in detecting novel object instances using models produced by weakly-supervised learning.
Multiple Hypothesis Tracking Revisited

This paper revisits the classical multiple hypotheses tracking (MHT) algorithm in a tracking-by-detection framework. The success of MHT largely depends on the ability to maintain a small list of potential hypotheses, which can be facilitated with the accurate object detectors that are currently available. We demonstrate that a classical MHT implementation from the 90's can come surprisingly close to the performance of state-of-the-art methods on standard benchmark datasets. In order to further utilize the strength of MHT in exploiting higher-order information, we introduce a method for training online appearance models for each track hypothesis. We show that appearance models can be learned efficiently via a regularized least squares framework, requiring only a few extra operations for each hypothesis branch. We obtain state-of-the-art results on popular tracking-by-detection datasets such as PETS and the recent MOT challenge.
Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture

Projection is the only viable way to produce very large displays. Rear projection of large-scale upright displays is often preferred over front projection because of the lack of shadows that occlude the projected image. However, rear projection is not always a feasible option for space and cost reasons. Recent research suggests that many of the desirable features of rear projection, in particular lack of shadows, can be reproduced using active virtual rear projection (VRP). We present a new approach to shadow detection that addresses limitations with previous work. Furthermore, we demonstrate how to exploit the image processing capabilities of a GPU to shift the main performance bottleneck from image processing to camera capture and projector display rates. The improvements presented in this paper enable a speed increase in image processing from 15Hz to 110Hz in our new active VRP prototype.
Spatially constrained segmentation of dermoscopy images

Dermoscopy is a technique used to better visualize pigmented skin lesion and aid the clinician in determining if a lesion is benign or malignant. Automated segmentation of dermoscopy images is an important step for computer-aided diagnosis of melanoma. In this paper, we investigate how to use the spatial constraints present in pigmented lesions to improve the segmentation of dermoscopy images. We present an unsupervised segmentation algorithm that embeds these constraints into the feature space. The algorithm groups image pixels with homogeneous properties, and merges the pixel groups into a few super-regions. The optimal lesion- skin boundary is chosen from the set of all region boundaries, where the optimality is determined from the color and texture properties of the regions. We test our method on 67 dermoscopy images and compare the automatically generated segmentation with dermatologist-determined segmentation. The results demonstrate the advantage of incorporating domain-specific constraints into the segmentation process.
A multiple hypothesis approach to figure tracking

This paper describes a probabilistic multiple-hypothesis framework for tracking highly articulated objects. In this framework, the probability density of the tracker state is represented as a set of modes with piecewise Gaussians characterizing the neighborhood around these modes. The temporal evolution of the probability density is achieved through sampling from the prior distribution, followed by local optimization of the sample positions to obtain updated modes. This method of generating hypotheses from state-space search does not require the use of discrete features unlike classical multiple-hypothesis tracking. The parametric form of the model is suited for high dimensional state-spaces which cannot be efficiently modeled using non-parametric approaches. Results are shown for tracking Fred Astaire in a movie dance sequence.
A flexible projector-camera system for multi-planar displays

We present a novel multi-planar display system based on an uncalibrated projector-camera pair. Our system exploits the juxtaposition of planar surfaces in a room to create ad-hoc visualization and display capabilities. In an office setting, for example, a desk pushed against a wall provides two perpendicular surfaces that can simultaneously display elevation and plan views of an architectural model. In contrast to previous room-level projector-camera systems, our method is based on a flexible calibration procedure that requires a minimum amount of information for the geometry of the multi-planar surface scenario. A number of display configurations can be created on any available planar surfaces using a single commodity projector and camera. The key to our calibration approach is an efficient technique for simultaneously localizing multiple planes and a robust planar metric rectification method, which can tolerate a restricted camera field-of-view and requires no special calibration objects. We demonstrate the robustness of our calibration method using real and synthetic images and present several applications of our display system.
Dynamic feature ordering for efficient registration

Existing sequential feature based registration algorithms involving search typically either select features randomly (e.g. the RANSAC approach (M. Fischler and R. Bolles, 1981)) or assume a predefined, intuitive ordering for the features (e.g. based on size or resolution). The paper presents a formal framework for computing an ordering for features which maximizes search efficiency. Features are ranked according to matching ambiguity measure, and an algorithm is proposed which couples the feature selection with the parameter estimation, resulting in a dynamic feature ordering. The analysis is extended to template features where the matching is non discrete and a sample refinement process is proposed. The framework is demonstrated effectively on the localization of a person in an image, using a kinematic model with template features. Different priors are used on the model parameters and the results demonstrate nontrivial variations in the optimal feature hierarchy.
Traversability classification for UGV navigation: a comparison of patch and superpixel representations

Robot navigation in complex outdoor terrain can benefit from accurate traversability classification. Appearance- based traversability estimation can provide a long-range sensing capability which complements the traditional use of stereo or LIDAR ranging. In the standard approach to traversability classification, each image frame is decomposed into patches or pixels for further analysis. However, classification at the pixel level is prone to noise and complicates the task of identifying homogeneous regions for navigation. Fixed-sized patches aggregate pixel information, resulting in better noise properties, but they can span multiple distinct image regions, which can degrade the classification performance and make thin obstacles difficult to detect. We address the use of superpixels as the visual primitives for traversability estimation. Superpixels are obtained from an over-segmentation of the image and they aggregate visually homogeneous pixels while respecting natural terrain boundaries. We show that superpixels are superior to patches in classification accuracy and result in more effective navigation in complex terrain environments. Our experimental results include a study of the effect of patch and superpixel size on classification accuracy. We demonstrate that superpixels can be computed on-line on a real robot at a sufficient frame rate to support long-range sensing and planning.
Automatic cascade training with perturbation bias

Face detection methods based on cascade architecture have demonstrated fast and robust performance. Cascade learning is aided by the modularity of the architecture in which nodes are chained together to form a cascade. In this paper we present two new cascade learning results which address the decoupled nature of the cascade learning task. First, we introduce a cascade indifference curve framework, which connects the learning objectives for a node to the overall cascade performance. We derive a new cost function for node learning, which yields fully-automatic stopping conditions and improved detection performance. Second, we introduce the concept of perturbation bias, which leverages the statistical differences between target and non-target classes in a detection problem to obtain improved performance and robustness. We derive necessary and sufficient conditions for the success of the method and present experimental results.
Guided pushing for object singulation

We propose a novel method for a robot to separate and segment objects in a cluttered tabletop environment. The method leverages the fact that external object boundaries produce visible edges within an object cluster. We achieve this singulation of objects by using the robot arm to perform pushing actions specifically selected to test whether particular visible edges correspond to object boundaries. We verify the separation of objects after a push by examining the clusters formed by geometric segmentation of regions residing on the table surface. To avoid explicitly representing and tracking edges across push behaviors we aggregate over all edges in a given orientation by representing the push-history as an orientation histogram. By tracking the history of directions pushed for each object cluster we can build evidence that a cluster cannot be further separated. We present quantitative and qualitative experimental results performed in a real home environment by a mobile manipulator using input from an RGB-D camera mounted on the robot's head. We show that our pushing strategy can more reliably obtain singulation in fewer pushes than an approach, that does not explicitly reason about boundary information.
Boosted audio-visual HMM for speech reading

We propose a new approach for combining acoustic and visual measurements to aid in recognizing lip shapes of a person speaking. Our method relies on computing the maximum likelihoods of (a) HMM used to model phonemes from the acoustic signal, and (b) HMM used to model visual features motions from video. One significant addition in this work is the dynamic analysis with features selected by AdaBoost, on the basis of their discriminant ability. This form of integration, leading to boosted HMM, permits AdaBoost to find the best features first, and then uses HMM to exploit dynamic information inherent in the signal.
Statistical color models with application to skin detection

The existence of large image datasets such as photos on the World Wide Web make it possible to build powerful generic models for low-level image attributes like color using simple histogram learning techniques. We describe the construction of color models for skin and non-skin classes from a dataset of nearly 1 billion labeled pixels. These classes exhibit a surprising degree of separability which we exploit by building a skin pixel detector that achieves an equal error rate of 88%. We compare the performance of histogram and mixture models in skin detection and find histogram models to be superior in accuracy and computational cost. Using aggregate features computed from the skin detector we build a remarkably effective detector for naked people. We believe this work is the most comprehensive and detailed exploration of skin color models to date.