OpenRaiser/Intern-Atlas huggingface dataset:
Fast mean shift by compact density representation
Authors: Daniel Freedman, Pavel Kisilev
Venue: Conference CVPR 2009
Year: 2009

The Mean Shift procedure is a well established clustering technique that is widely used in imaging applications such as image and video segmentation, denoising, object tracking, texture classification, and others. However, the Mean Shift procedure has relatively high time complexity which is superlinear in the number of data points. In this paper we present a novel fast Mean Shift procedure which is based on the random sampling of the Kernel Density Estimate (KDE). We show theoretically that the resulting reduced KDE is close to the complete data KDE, to within a given accuracy. Moreover, we prove that the time complexity of the proposed fast Mean Shift procedure based on the reduced KDE is considerably lower than that of the original Mean Shift; the typical gain is of several orders for big data sets. Experiments show that image and video segmentation results of the proposed fast Mean Shift method are similar to those based on the standard Mean shift procedure. We also present a new application of the Fast Mean Shift method to the efficient construction of graph hierarchies for images; the resulting structure is potentially useful for solving computer vision problems which can be posed as graph problems, including stereo, semi-automatic segmentation, and optical flow.
Shop the look: Building a large scale visual shopping system at Pinterest
Authors: Raymond Shiau, Hao-Yu Wu, Eric Kim, Yue Li Du, Anqi Guo, Zhiyuan Zhang, Eileen Li, Kunlong Gu, Charles Rosenberg 0001, Andrew Zhai
Venue: Conference KDD 2020
Year: 2020

As online content becomes ever more visual, the demand for searching by visual queries grows correspondingly stronger. Shop The Look is an online shopping discovery service at Pinterest, leveraging visual search to enable users to find and buy products within an image. In this work, we provide a holistic view of how we built Shop The Look, a shopping oriented visual search system, along with lessons learned from addressing shopping needs. We discuss topics including core technology across object detection and visual embeddings, serving infrastructure for realtime inference, and data labeling methodology for training/evaluation data collection and human evaluation. The user-facing impacts of our system design choices are measured through offline evaluations, human relevance judgements, and online A/B experiments. The collective improvements amount to cumulative relative gains of over 160% in end-to-end human relevance judgements and over 80% in engagement. Shop The Look is deployed in production at Pinterest.
Trajectory triangulation over conic sections
Authors: Amnon Shashua, Shai Avidan, Michael Werman
Venue: Conference ICCV 1999
Year: 1999

We consider the problem of reconstructing the 3D coordinates of a moving point seen from a monocular moving camera, i.e., to reconstruct moving objects from line-of-sight measurements only. The task is feasible only when same constraints are placed on the shape of the trajectory of the moving point. We coin the family of such tasks as "trajectory triangulation". In this paper we focus on trajectories whose shape is a conic-section and show that generally 9 views are sufficient for a unique reconstruction of the moving point and fewer views when the conic is a known type (like a circle in 3D Euclidean space for which 7 views are sufficient). Experiments demonstrate that our solutions are practical. The paradigm of Trajectory Triangulation in general pushes the envelope of processing dynamic scenes forward. Thus static scenes become a particular case of a more general task of reconstructing scenes rich with moving objects (where an object could be a single point).
Automated articulated structure and 3D shape recovery from point correspondences
Authors: João Fayad, Chris Russell 0001, Lourdes Agapito
Venue: Conference ICCV 2011
Year: 2011

In this paper we propose a new method for the simultaneous segmentation and 3D reconstruction of interest point based articulated motion. We decompose a set of point tracks into rigid-bodied overlapping regions which are associated with skeletal links, while joint centres can be derived from the regions of overlap. This allows us to formulate the problem of 3D reconstruction as one of model assignment, where each model corresponds to the motion and shape parameters of an articulated body part. We show how this labelling can be optimised using a combination of pre-existing graph-cut based inference, and robust structure from motion factorization techniques. The strength of our approach comes from viewing both the decomposition into parts, and the 3D reconstruction as the optimisation of a single cost function, namely the image re-projection error. We show results of full 3D shape recovery on challenging real-world sequences with one or more articulated bodies, in the presence of outliers and missing data.
Two-view motion segmentation by mixtures of Dirichlet process with model selection and outlier removal
Authors: Yong-Dian Jian, Chu-Song Chen
Venue: Conference ICCV 2007
Year: 2007

This paper presents a novel motion segmentation algorithm on the basis of mixture of Dirichlet process (MDP) models, a kind of nonparametric Bayesian framework. In contrast to previous approaches, our method consider motion segmentation and its model selection regarding to the number of motion models as an indivisible problem. The proposed algorithm can simultaneously infer the number of motion models, estimate the cluster memberships of correspondence points, and identify the outliers of input data. The key idea is to use MDP models to fully exploit the epipolar constraints before making premature decisions about the number motion models. To handle outliers efficiently, we then incorporate RANSAC within the inference process of MDP models and make them take the advantages of each other. In the experiments, we compare the proposed algorithm with naive RANSAC, GPCA and Schindler's method on both synthetic data and real image data. The experimental results show that we can handle more motions and still have satisfactory performance in the presence of various levels of noise and outlier.
Automatic reconstruction of piecewise planar models from multiple views
Authors: Caroline Baillard, Andrew Zisserman
Venue: Conference CVPR 1999
Year: 1999

A new method is described for automatically reconstructing 3D planar faces from multiple images of a scene. The novelty of the approach lies in the use of inter-image homographies to validate and best estimate the plane, and in the minimal initialization requirements-only a single 3D line with a textured neighbourhood is required to generate a plane hypothesis. The planar facets enable line grouping and also the construction of parts of the wireframe which were missed due to the inevitable shortcomings of feature detection and matching. The method allows a piecewise planar model of a scene to be built completely automatically, with no user intervention at any stage, given only the images and camera projection matrices as input. The robustness and reliability of the method are illustrated on several examples, from both aerial and interior views.
Weakly supervised discriminative localization and classification: a joint learning process
Authors: Minh Hoai Nguyen, Lorenzo Torresani, Lorenzo de la Torre, Carsten Rother
Venue: Conference ICCV 2009
Year: 2009

Visual categorization problems, such as object classification or action recognition, are increasingly often approached using a detection strategy: a classifier function is first applied to candidate subwindows of the image or the video, and then the maximum classifier score is used for class decision. Traditionally, the subwindow classifiers are trained on a large collection of examples manually annotated with masks or bounding boxes. The reliance on time-consuming human labeling effectively limits the application of these methods to problems involving very few categories. Furthermore, the human selection of the masks introduces arbitrary biases (e.g. in terms of window size and location) which may be suboptimal for classification. In this paper we propose a novel method for learning a discriminative subwindow classifier from examples annotated with binary labels indicating the presence of an object or action of interest, but not its location. During training, our approach simultaneously localizes the instances of the positive class and learns a subwindow SVM to recognize them. We extend our method to classification of time series by presenting an algorithm that localizes the most discriminative set of temporal segments in the signal. We evaluate our approach on several datasets for object and action recognition and show that it achieves results similar and in many cases superior to those obtained with full supervision.
Sigma set: A small second order statistical region descriptor
Authors: Xiaopeng Hong, Hong Chang 0001, Shiguang Shan, Xilin Chen 0001, Wen Gao 0001
Venue: Conference CVPR 2009
Year: 2009

Given an image region of pixels, second order statistics can be used to construct a descriptor for object representation. One example is the covariance matrix descriptor, which shows high discriminative power and good robustness in many computer vision applications. However, operations for the covariance matrix on Riemannian manifolds are usually computationally demanding. This paper proposes a novel second order statistics based region descriptor, named “Sigma Set”, in the form of a small set of vectors, which can be uniquely constructed through Cholesky decomposition on the covariance matrix. Sigma Set is of low dimension, powerful and robust. Moreover, compared with the covariance matrix, Sigma Set is not only more efficient in distance evaluation and average calculation, but also easier to be enriched with first order statistics. Experimental results in texture classification and object tracking verify the effectiveness and efficiency of this novel object descriptor.
Talking face generation based on information bottleneck and complementary representations
Authors: Jie Tang, Yiling Wu, Minglei Li, Zhu Wang
Venue: Conference CIKM 2021
Year: 2021

Audio-driven talking face generation is an active research direction in the field of virtual reality. The main challenge is that the generated lip shape of the speaker is out of sync with the input audio. To address this challenge, we propose a novel solution to synthesize lip-synchronized, high-quality, realistic video given input audio. We first decompose the target person's video frames into 3D face model parameters, and the information bottleneck is inserted into the audio-to-expression network to learn the mapping between audio features and expression parameters. Then, we replace the expression parameters in the target video frame with the extracted expression parameters from audio and re-render the face. Finally, we add high-level audio embedding extracted from the raw audio and lip landmarks embedding in the neural rendering network. The 3D face shapes, 2D landmarks, and audio embedding provide complementary information for the neural rendering network which guarantees the generation of lip-synchronized high-quality video portraits from the synthesized rendered faces. Experimental results show that compared with other talking face generation methods, our method is the best concerning lip synchronization with high video definition.
Linear view synthesis using a dimensionality gap light field prior
Authors: Anat Levin, Frédo Durand
Venue: Conference CVPR 2010
Year: 2010

Acquiring and representing the 4D space of rays in the world (the light field) is important for many computer vision and graphics applications. Yet, light field acquisition is costly due to their high dimensionality. Existing approaches either capture the 4D space explicitly, or involve an error-sensitive depth estimation process. This paper argues that the fundamental difference between different acquisition and rendering techniques is a difference between prior assumptions on the light field. We use the previously reported dimensionality gap in the 4D light field spectrum to propose a new light field prior. The new prior is a Gaussian assigning a non-zero variance mostly to a 3D subset of entries. Since there is only a low-dimensional subset of entries with non-zero variance, we can reduce the complexity of the acquisition process and render the 4D light field from 3D measurement sets. Moreover, the Gaussian nature of the prior leads to linear and depth invariant reconstruction algorithms. We use the new prior to render the 4D light field from a 3D focal stack sequence and to interpolate sparse directional samples and aliased spatial measurements. In all cases the algorithm reduces to a simple spatially invariant deconvolution which does not involve depth estimation.
Segmenting visual actions based on spatio-temporal motion patterns
Authors: Yong Rui, P. Anandan 0001
Venue: Conference CVPR 2000
Year: 2000

The analysis of human action captured in video sequences has been a topic of considerable interest in computer vision. Much of the previous work has focused on the problem of action or activity recognition, but ignored the problem of detecting action boundaries in a video sequence containing unfamiliar and arbitrary visual actions. This paper presents an approach to this problem based on detecting temporal discontinuities of the spatial pattern of image motion that captures the action. We represent frame to frame optical-flow in terms of the coefficients of the most significant principal components computed from all the flow-fields within a given video sequence. We then detect the discontinuities in the temporal trajectories of these coefficients based on three different measures. We compare our segment boundaries against those detected by human observers on the same sequences in a recent independent psychological study of human perception of visual events. We show experimental results on the two sequences that were used in this study. Our experimental results are promising both from visual evaluation and when compared against the results of the psychological study.
High quality mesostructure acquisition using specularities
Authors: Yannick Francken, Tom Cuypers, Tom Mertens, Jo Gielis, Philippe Bekaert
Venue: Conference CVPR 2008
Year: 2008

We propose a technique for cheap and efficient acquisition of mesostructure normal maps from specularities, which only requires a simple LCD monitor and a digital camera. Coded illumination enables us to capture subtle surface details with only a handful of images. In addition, our method can deal with heterogeneous surfaces, and high albedo materials. We are able to recover highly detailed mesostructures, which was previously only possible with an expensive hardware setup.
Probabilistic image registration and anomaly detection by nonlinear warping
Authors: Verena Kaynig, Bernd Fischer 0003, Joachim M. Buhmann
Venue: Conference CVPR 2008
Year: 2008

Automatic, defect tolerant registration of transmission electron microscopy (TEM) images poses an important and challenging problem for biomedical image analysis, e.g. in computational neuroanatomy. In this paper we demonstrate a fully automatic stitching and distortion correction method for TEM images and propose a probabilistic approach for image registration. The technique identifies image defects due to sample preparation and image acquisition by outlier detection. A polynomial kernel expansion is used to estimate a non-linear image transformation based on intensities and spatial features. Corresponding points in the images are not determined beforehand, but they are estimated via an EM-algorithm during the registration process which is preferable in the case of (noisy) TEM images. Our registration model is successfully applied to two large image stacks of serial section TEM images acquired from brain tissue samples in a computational neuroanatomy project and shows significant improvement over existing image registration methods on these large datasets.
The von kries hypothesis and a basis for color constancy
Authors: Hamilton Y. Chong, Steven J. Gortler, Todd E. Zickler
Venue: Conference ICCV 2007
Year: 2007

Color constancy is almost exclusively modeled with diagonal transforms. However, the choice of basis under which diagonal transforms are taken is traditionally ad hoc. Attempts to remedy the situation have been hindered by the fact that no joint characterization of the conditions for {sensors, illuminants, reflectances} to support diagonal color constancy has previously been achieved. In this work, we observe that the von Kries compatibility conditions are impositions only on the sensor measurements, not the physical spectra. This allows us to formulate the von Kries compatibility conditions succinctly as rank constraints on an order 3 measurement tensor. Given this, we propose an algorithm that computes a (locally) optimal choice of color basis for diagonal color constancy and compare the results against other proposed choices.
Gating dropout: Communication-efficient regularization for sparsely activated transformers
Authors: Rui Liu 0013, Young Jin Kim 0001, Alexandre Muzio, Hany Hassan
Venue: Conference ICML 2022
Year: 2022

Sparsely activated transformers, such as Mixture of Experts (MoE), have received great interest due to their outrageous scaling capability which enables dramatical increases in model size without significant increases in computational cost. To achieve this, MoE models replace the feedforward sub-layer with Mixture-of-Experts sub-layer in transformers and use a gating network to route each token to its assigned experts. Since the common practice for efficient training of such models requires distributing experts and tokens across different machines, this routing strategy often incurs huge cross-machine communication cost because tokens and their assigned experts likely reside in different machines. In this paper, we propose \emph{Gating Dropout}, which allows tokens to ignore the gating network and stay at their local machines, thus reducing the cross-machine communication. Similar to traditional dropout, we also show that Gating Dropout has a regularization effect during training, resulting in improved generalization performance. We validate the effectiveness of Gating Dropout on multilingual machine translation tasks. Our results demonstrate that Gating Dropout improves a state-of-the-art MoE model with faster wall-clock time convergence rates and better BLEU scores for a variety of model sizes and datasets.
Discovering and exploiting 3d symmetries in structure from motion
Authors: Andrea Cohen, Christopher Zach, Sudipta N. Sinha, Marc Pollefeys
Venue: Conference CVPR 2012
Year: 2012

Many architectural scenes contain symmetric or repeated structures, which can generate erroneous image correspondences during structure from motion (Sfm) computation. Prior work has shown that the detection and removal of these incorrect matches is crucial for accurate and robust recovery of scene structure. In this paper, we point out that these incorrect matches, in fact, provide strong cues to the existence of symmetries and structural regularities in the unknown 3D structure. We make two key contributions. First, we propose a method to recover various symmetry relations in the structure using geometric and appearance cues. A set of structural constraints derived from the symmetries are imposed within a new constrained bundle adjustment formulation, where symmetry priors are also incorporated. Second, we show that the recovered symmetries enable us to choose a natural coordinate system for the 3D structure where gauge freedom in rotation is held fixed. Furthermore, based on the symmetries, 3D structure completion is also performed. Our approach significantly reduces drift through ”structural” loop closures and improves the accuracy of reconstructions in urban scenes.
Randomized structure from motion based on atomic 3d models from camera triplets
Authors: Michal Havlena, Akihiko Torii, Jan Knopp, Tomás Pajdla
Venue: Conference CVPR 2009
Year: 2009

This paper presents a new efficient technique for large-scale structure from motion from unordered data sets. We avoid costly computation of all pairwise matches and geometries by sampling pairs of images using the pairwise similarity scores based on the detected occurrences of visual words leading to a significant speedup. Furthermore, atomic 3D models reconstructed from camera triplets are used as the seeds which form the final large-scale 3D model when merged together. Using three views instead of two allows us to reveal most of the outliers of pairwise geometries at an early stage of the process hindering them from derogating the quality of the resulting 3D structure at later stages. The accuracy of the proposed technique is shown on a set of 64 images where the result of the exhaustive technique is known. Scalability is demonstrated on a landmark reconstruction from hundreds of images.
Range image segmentation by surface extraction using an improved robust estimator
Authors: Paulo F. U. Gotardo, Olga Regina Pereira Bellon, Luciano Silva
Venue: Conference CVPR 2003
Year: 2003

The paper presents a novel range image segmentation algorithm based on planar surface extraction. The algorithm was applied to common range image databases and was favorably compared against seven other segmentation algorithms using a popular evaluation framework. The experimental results show that, as compared to the other methods, our algorithm presents a good performance in preserving small regions and edge locations when processing noisy images. Our main contribution is an improved robust estimator, derived from the RANSAC and MSAC estimators, whose optimization process is accelerated by a genetic algorithm with a new set of parameters and operations designed to avoid premature convergence.
Building large urban environments from unstructured point data
Authors: Florent Lafarge, Clément Mallet
Venue: Conference ICCV 2011
Year: 2011

We present a robust method for modeling cities from unstructured point data. Our algorithm provides a more complete description than existing approaches by reconstructing simultaneously buildings, trees and topologically complex grounds. Buildings are modeled by an original approach which guarantees a high generalization level while having semantized and compact representations. Geometric 3D-primitives such as planes, cylinders, spheres or cones describe regular roof sections, and are combined with mesh-patches that represent irregular roof components. The various urban components interact through a non-convex energy minimization problem in which they are propagated under arrangement constraints over a planimetric map. We experimentally validate the approach on complex urban structures and large urban scenes of millions of points.
Critical configurations for n-view projective reconstruction
Authors: Fredrik Kahl, Richard I. Hartley, Kalle Åström
Venue: Conference CVPR 2001
Year: 2001

In this paper we give a characterization of critical configurations for projective reconstruction with any number of points and views. A set of cameras and points is said to be critical if the projected image points are insufficient to determine the placement of the points and the cameras uniquely, up to a projective transformation. For two views, the critical configurations are well-known. In this paper it is shown that a configuration of n 3 cameras and in points all lying on the intersection of two distinct ruled quadrics is critical. In distinction to the two-view case, which in general allows two alternative solutions, there is a family of ambiguous reconstructions for the n-view case. As a partial converse, it Is shown that for any critical configuration, all the points lie on the intersection of two ruled quadrics.