OpenRaiser/Intern-Atlas huggingface dataset:
Connected segmentation tree – a joint representation of region layout and hierarchy
Authors: Narendra Ahuja, Sinisa Todorovic
Venue: Conference CVPR 2008
Year: 2008

This paper proposes a new object representation, called connected segmentation tree (CST), which captures canonical characteristics of the object in terms of the photometric, geometric, and spatial adjacency and containment properties of its constituent image regions. CST is obtained by augmenting the objectpsilas segmentation tree (ST) with inter-region neighbor links, in addition to their recursive embedding structure already present in ST. This makes CST a hierarchy of region adjacency graphs. A regionpsilas neighbors are computed using an extension to regions of the Voronoi diagram for point patterns. Unsupervised learning of the CST model of a category is formulated as matching the CST graph representations of unlabeled training images, and fusing their maximally matching subgraphs. A new learning algorithm is proposed that optimizes the model structure by simultaneously searching for both the most salient nodes (regions) and the most salient edges (containment and neighbor relationships of regions) across the image graphs. Matching of the category model to the CST of a new image results in simultaneous detection, segmentation and recognition of all occurrences of the category, and a semantic explanation of these results.
Simultaneous localization, mapping and deblurring
Authors: Hee Seok Lee, Junghyun Kwon, Kyoung Mu Lee
Venue: Conference ICCV 2011
Year: 2011

Handling motion blur is one of important issues in visual SLAM. For a fast-moving camera, motion blur is an unavoidable effect and it can degrade the results of localization and reconstruction severely. In this paper, we present a unified algorithm to handle motion blur for visual SLAM, including the blur-robust data association method and the fast deblurring method. In our framework, camera motion and 3-D point structures are reconstructed by SLAM, and the information from SLAM makes the estimation of motion blur quite easy and effective. Reversely, estimating motion blur enables robust data association and drift-free localization of SLAM with blurred images. The blurred images are recovered by fast deconvolution using SLAM data, and more features are extracted and registered to the map so that the SLAM procedure can be continued even with the blurred images. In this way, visual SLAM and deblurring are solved simultaneously, and improve each other's results significantly.
Modeling and generating complex motion blur for real-time tracking
Authors: Christopher Mei, Ian D. Reid 0001
Venue: Conference CVPR 2008
Year: 2008

This article addresses the problem of real-time visual tracking in presence of complex motion blur. Previous authors have observed that efficient tracking can be obtained by matching blurred images instead of applying the computationally expensive task of deblurring (H. Jin et al., 2005). The study was however limited to translational blur. In this work, we analyse the problem of tracking in presence of spatially variant motion blur generated by a planar template. We detail how to model the blur formation and parallelise the blur generation, enabling a real-time GPU implementation. Through the estimation of the camera exposure time, we discuss how tracking initialisation can be improved. Our algorithm is tested on challenging real data with complex motion blur where simple models fail. The benefit of blur estimation is shown for structure and motion.
Dense 3d motion capture from synchronized video streams
Authors: Yasutaka Furukawa, Jean Ponce
Venue: Conference CVPR 2008
Year: 2008

This paper proposes a novel approach to non-rigid, markerless motion capture from synchronized video streams acquired by calibrated cameras. The instantaneous geometry of the observed scene is represented by a polyhedral mesh with fixed topology. The initial mesh is constructed in the first frame using the publicly available PMVS software for multi-view stereo [7]. Its deformation is captured by tracking its vertices over time, using two optimization processes at each frame: a local one using a rigid motion model in the neighborhood of each vertex, and a global one using a regularized nonrigid model for the whole mesh. Qualitative and quantitative experiments using seven real datasets show that our algorithm effectively handles complex nonrigid motions and severe occlusions.
Fine-grained patient similarity measuring using deep metric learning
Authors: Jiazhi Ni, Jie Liu, Chenxin Zhang, Dan Ye, Zhirou Ma
Venue: Conference CIKM 2017
Year: 2017

Patient similarity measuring plays a significant role in many healthcare applications, such as cohort study and treatment comparative effectiveness research. Existing methods mainly rely on supervised metric learning method to study patient similarity from Electronic Health Records (EHRs), facing the challenge of differentiating patients with a large number of fine-grained disease categories. Deep metric learning has gained noticeable success in fine-grained image categorization problem, however, it cannot be directly applied to classification of patients with hierarchical disease labels. In this paper, we present a novel three layer patient similarity deep metric learning framework (PSDML) by optimizing quadruple loss improved from triplet loss, to learn an embedding distance for disease classification among the patients. The context semantic relation of multi diagnosis labels encoding by ICD-10 is taken into account to compute the supervised distance of patients. To solve the diagnosis class imbalance, patient tuples that violate deep metric learning framework loss constraints are chosen prior as samples to accelerate the convergence of the neural network. We conducted KNN multi label classification experiment using the learned similarity metric on the real EHRs about stroke disease collected by Chinese Stroke Data Center. The results demonstrate substantial improvement over the baselines.
Talking pictures: Temporal grouping and dialog-supervised person recognition
Authors: Timothée Cour, Benjamin Sapp, Akash Nagle, Ben Taskar
Venue: Conference CVPR 2010
Year: 2010

We address the character identification problem in movies and television videos: assigning names to faces on the screen. Most prior work on person recognition in video assumes some supervised data such as screenplay or handlabeled faces. In this paper, our only source of `supervision' are the dialog cues: first, second and third person references (such as “I'm Jack”, “Hey, Jack!” and “Jack left”). While this kind of supervision is sparse and indirect, we exploit multiple modalities and their interactions (appearance, dialog, mouth movement, synchrony, continuity-editing cues) to effectively resolve identities through local temporal grouping followed by global weakly supervised recognition. We propose a novel temporal grouping model that partitions face tracks across multiple shots while respecting appearance, geometric and film-editing cues and constraints. In this model, states represent partitions of the k most recent face tracks, and transitions represent compatibility of consecutive partitions. We present dynamic programming inference and discriminative learning for the model. The individual face tracks are subsequently assigned a name by learning a classifier from partial label constraints. The weakly supervised classifier incorporates multiple-instance constraints from dialog cues as well as soft grouping constraints from our temporal grouping. We evaluate both the temporal grouping and final character naming on several hours of TV and movies.
Layered object detection for multi-class segmentation
Authors: Yi Yang 0007, Sam Hallman, Deva Ramanan, Charless C. Fowlkes
Venue: Conference CVPR 2010
Year: 2010

We formulate a layered model for object detection and multi-class segmentation. Our system uses the output of a bank of object detectors in order to define shape priors for support masks and then estimates appearance, depth ordering and labeling of pixels in the image. We train our system on the PASCAL segmentation challenge dataset and show good test results with state of the art performance in several categories including segmenting humans.
Facial expression editing in video using a temporally-smooth factorization
Authors: Fei Yang 0001, Lubomir D. Bourdev, Eli Shechtman, Jue Wang 0001, Dimitris N. Metaxas
Venue: Conference CVPR 2012
Year: 2012

We address the problem of editing facial expression in video, such as exaggerating, attenuating or replacing the expression with a different one in some parts of the video. To achieve this we develop a tensor-based 3D face geometry reconstruction method, which fits a 3D model for each video frame, with the constraint that all models have the same identity and requiring temporal continuity of pose and expression. With the identity constraint, the differences between the underlying 3D shapes capture only changes in expression and pose. We show that various expression editing tasks in video can be achieved by combining face reordering with face warping, where the warp is induced by projecting differences in 3D face shapes into the image plane. Analogously, we show how the identity can be manipulated while fixing expression and pose. Experimental results show that our method can effectively edit expressions and identity in video in a temporally-coherent way with high fidelity.
Time-critical search
Authors: Nina Mishra, Ryen W. White, Samuel Ieong, Eric Horvitz
Venue: Conference SIGIR 2014
Year: 2014

We study time-critical search, where users have urgent information needs in the context of an acute problem. As examples, users may need to know how to stem a severe bleed, help a baby who is choking on a foreign object, or respond to an epileptic seizure. While time-critical situations and actions have been studied in the realm of decision-support systems, little has been done with time-critical search and retrieval, and little direct support is offered by search systems. Critical challenges with time-critical search include accurately inferring when users have urgent needs and providing relevant information that can be understood and acted upon quickly. We leverage surveys and search log data from a large mobile search provider to (a) characterize the use of search engines for time-critical situations, and (b) develop predictive models to accurately predict urgent information needs, given a query and a diverse set of features spanning topical, temporal, behavioral, and geospatial attributes. The methods and findings highlight opportunities for extending search and retrieval to consider the urgency of queries.
Bayesian human segmentation in crowded situations
Authors: Tao Zhao 0001, Ramakant Nevatia
Venue: Conference CVPR 2003
Year: 2003

The problem of segmenting individual humans in crowded situations from stationary video camera sequences is exacerbated by object inter-occlusion. We pose this problem as a "model-based segmentation" problem in which human shape models are used to interpret the foreground in a Bayesian framework. The solution is obtained by using an efficient Markov chain Monte Carlo (MCMC) method that uses domain knowledge as proposal probabilities. Knowledge of various aspects including human shape, human height, camera model, and image cues including human head candidates, foreground/background separation are integrated in one theoretically sound framework. We show promising results and evaluations on some challenging data.
An eye for an eye: A single camera gaze-replacement method
Authors: Lior Wolf, Ziv Freund, Shai Avidan
Venue: Conference CVPR 2010
Year: 2010

The camera in video conference systems is typically positioned above, or below, the screen, causing the gaze of the users to appear misplaced. We propose an effective solution to this problem that is based on replacing the eyes of the user. This replacement, when done accurately, is enough to achieve a natural looking video. At an initialization stage the user is asked to look straight at the camera. We store these frames, then track the eyes accurately in the video sequence and replace the eyes, taking care of illumination and ghosting artifacts. We have tested the system on a large number of videos demonstrating the effectiveness of the proposed solution.
Adaptive fragments-based tracking of non-rigid objects using level sets
Authors: Prakash Chockalingam, S. Nalin Pradeep, Stan Birchfield
Venue: Conference ICCV 2009
Year: 2009

We present an approach to visual tracking based on dividing a target into multiple regions, or fragments. The target is represented by a Gaussian mixture model in a joint feature-spatial space, with each ellipsoid corresponding to a different fragment. The fragments are automatically adapted to the image data, being selected by an efficient region-growing procedure and updated according to a weighted average of the past and present image statistics. Modeling of target and background are performed in a Chan-Vese manner, using the framework of level sets to preserve accurate boundaries of the target. The extracted target boundaries are used to learn the dynamic shape of the target over time, enabling tracking to continue under total occlusion. Experimental results on a number of challenging sequences demonstrate the effectiveness of the technique.
A probabilistic framework for space carving
Authors: Adrian Broadhurst, Tom Drummond, Roberto Cipolla
Venue: Conference ICCV 2001
Year: 2001

This paper introduces a new probabilistic framework for Space Carving. In this framework each voxel is assigned a probability, which is computed by comparing the likelihoods for the voxel existing and not existing. This new framework avoids many of the difficulties associated with the original Space Carving algorithm. Specifically, it does not need a global threshold parameter, and it guarantees that no holes will be carved in the model. This paper also proposes that a voxel-based thick texture is a realistic and efficient representation for scenes which contain dominant planes. The algorithm is tested using both real and synthetic data, and both qualitative and quantitative results are presented.
Recognition by association via learning per-exemplar distances
Authors: Tomasz Malisiewicz, Alexei A. Efros
Venue: Conference CVPR 2008
Year: 2008

We pose the recognition problem as data association. In this setting, a novel object is explained solely in terms of a small set of exemplar objects to which it is visually similar. Inspired by the work of Frome et al., we learn separate distance functions for each exemplar; however, our distances are interpretable on an absolute scale and can be thresholded to detect the presence of an object. Our exemplars are represented as image regions and the learned distances capture the relative importance of shape, color, texture, and position features for that region. We use the distance functions to detect and segment objects in novel images by associating the bottom-up segments obtained from multiple image segmentations with the exemplar regions. We evaluate the detection and segmentation performance of our algorithm on real-world outdoor scenes from the LabelMe (B. Russel, et al., 2007) dataset and also show some promising qualitative image parsing results.
Jointly modeling static visual appearance and temporal pattern for unsupervised video hashing
Authors: Chao Li, Yang Yang, Jiewei Cao, Zi Huang
Venue: Conference CIKM 2017
Year: 2017

Recently, hashing has been evidenced as an efficient and effective method to facilitate large-scale video retrieval. Most of existing hashing methods are based on visual features, which are expected to capture the appearance of videos. The intrinsic temporal pattern embedded in videos has also shown its discriminative power for similarity search, and is explored and utilised in some recent studies. However, how to leverage the strengths in both aspects remains unknown.
Action snippets: How many frames does human action recognition require?
Authors: Konrad Schindler, Luc Van Gool
Venue: Conference CVPR 2008
Year: 2008

Visual recognition of human actions in video clips has been an active field of research in recent years. However, most published methods either analyse an entire video and assign it a single action label, or use relatively large look-ahead to classify each frame. Contrary to these strategies, human vision proves that simple actions can be recognised almost instantaneously. In this paper, we present a system for action recognition from very short sequences (ldquosnippetsrdquo) of 1-10 frames, and systematically evaluate it on standard data sets. It turns out that even local shape and optic flow for a single frame are enough to achieve ap90% correct recognitions, and snippets of 5-7 frames (0.3-0.5 seconds of video) are enough to achieve a performance similar to the one obtainable with the entire video sequence.
Local trinary patterns for human action recognition
Authors: Lahav Yeffet, Lior Wolf
Venue: Conference ICCV 2009
Year: 2009

We present a novel action recognition method which is based on combining the effective description properties of Local Binary Patterns with the appearance invariance and adaptability of patch matching based methods. The resulting method is extremely efficient, and thus is suitable for real-time uses of simultaneous recovery of human action of several lengths and starting points. Tested on all publicity available datasets in the literature known to us, our system repeatedly achieves state of the art performance. Lastly, we present a new benchmark that focuses on uncut motion recognition in broadcast sports video.
Simultaneous super-resolution and 3d video using graph-cuts
Authors: Tony Tung, Shohei Nobuhara, Takashi Matsuyama
Venue: Conference CVPR 2008
Year: 2008

This paper presents a new method to increase the quality of 3D video, a new media developed to represent 3D objects in motion. This representation is obtained from multi-view reconstruction techniques that require images recorded simultaneously by several video cameras. All cameras are calibrated and placed around a dedicated studio to fully surround the models. The limited quality and quantity of cameras may produce inaccurate 3D model reconstruction with low quality texture. To overcome this issue, first we propose super-resolution (SR) techniques for 3D video: SR on multi-view images and SR on single-view video frames. Second, we propose to combine both super-resolution and dynamic 3D shape reconstruction problems into a unique Markov Random Field (MRF) energy formulation. The MRF minimization is performed using graph-cuts. Thus, we jointly compute the optimal solution for super-resolved texture and 3D shape model reconstruction. Moreover, we propose a coarse-to-fine strategy to iteratively produce 3D video with increasing quality. Our experiments show the accuracy and robustness of the proposed technique on challenging 3D video sequences.
Complete multi-view reconstruction of dynamic scenes from probabilistic fusion of narrow and wide baseline stereo
Authors: Tony Tung, Shohei Nobuhara, Takashi Matsuyama
Venue: Conference ICCV 2009
Year: 2009

This paper presents a novel approach to achieve accurate and complete multi-view reconstruction of dynamic scenes (or 3D videos). 3D videos consist in sequences of 3D models in motion captured by a surrounding set of video cameras. To date 3D videos are reconstructed using multiview wide baseline stereo (MVS) reconstruction techniques. However it is still tedious to solve stereo correspondence problems: reconstruction accuracy falls when stereo photo-consistency is weak, and completeness is limited by self-occlusions. Most MVS techniques were indeed designed to deal with static objects in a controlled environment and therefore cannot solve these issues. Hence we propose to take advantage of the image content stability provided by each single-view video to recover any surface regions visible by at least one camera. In particular we present an original probabilistic framework to derive and predict the true surface of models. We propose to fuse multi-view structure-from-motion with robust 3D features obtained by MVS in order to significantly improve reconstruction completeness and accuracy. A min-cut problem where all exact features serve as priors is solved in a final step to reconstruct the 3D models. In addition, experimental results were conducted on synthetic and challenging real world datasets to illustrate the robustness and accuracy of our method.
Data-driven crowd analysis in videos
Authors: Mikel Rodriguez, Josef Sivic, Ivan Laptev, Jean-Yves Audibert
Venue: Conference ICCV 2011
Year: 2011

In this work we present a new crowd analysis algorithm powered by behavior priors that are learned on a large database of crowd videos gathered from the Internet. The algorithm works by first learning a set of crowd behavior priors off-line. During testing, crowd patches are matched to the database and behavior priors are transferred. We adhere to the insight that despite the fact that the entire space of possible crowd behaviors is infinite, the space of distinguishable crowd motion patterns may not be all that large. For many individuals in a crowd, we are able to find analogous crowd patches in our database which contain similar patterns of behavior that can effectively act as priors to constrain the difficult task of tracking an individual in a crowd. Our algorithm is data-driven and, unlike some crowd characterization methods, does not require us to have seen the test video beforehand. It performs like state-of-the-art methods for tracking people having common crowd behaviors and outperforms the methods when the tracked individual behaves in an unusual way.
Support Vector Regression for Multi-View Gait Recognition based on Local Motion Feature Selection
Authors: Worapan Kusakunniran, Qiang Wu 0001, Jian Zhang 0002, Hongdong Li
Venue: Conference CVPR 2010
Year: 2010

Gait is a well recognized biometric feature that is used to identify a human at a distance. However, in real environment, appearance changes of individuals due to viewing angle changes cause many difficulties for gait recognition. This paper re-formulates this problem as a regression problem. A novel solution is proposed to create a View Transformation Model (VTM) from the different point of view using Support Vector Regression (SVR). To facilitate the process of regression, a new method is proposed to seek local Region of Interest (ROI) under one viewing angle for predicting the corresponding motion information under another viewing angle. Thus, the well constructed VTM is able to transfer gait information under one viewing angle into another viewing angle. This proposal can achieve view-independent gait recognition. It normalizes gait features under various viewing angles into a common viewing angle before similarity measurement is carried out. The extensive experimental results based on widely adopted benchmark dataset demonstrate that the proposed algorithm can achieve significantly better performance than the existing methods in literature.
Aligning ASL for statistical translation using a discriminative word model
Authors: Ali Farhadi, David A. Forsyth
Venue: Conference CVPR 2006
Year: 2006

We describe a method to align ASL video subtitles with a closed-caption transcript. Our alignments are partial, based on spotting words within the video sequence, which consists of joined (rather than isolated) signs with unknown word boundaries. We start with windows known to contain an example of a word, but not limited to it. We estimate the start and end of the word in these examples using a voting method. This provides a small number of training examples (typically three per word). Since there is no shared structure, we use a discriminative rather than a generative word model. While our word spotters are not perfect, they are sufficient to establish an alignment. We demonstrate that quite small numbers of good word spotters results in an alignment good enough to produce simple English-ASL translations, both by phrase matching and using word substitution.