OpenRaiser/Intern-Atlas huggingface dataset:
Learning bilinear models for two-factor problems in vision
Authors: William T. Freeman, Joshua B. Tenenbaum
Venue: Conference CVPR 1997
Year: 1997

In many vision problems, we want to infer two (or more) hidden factors which interact to produce our observations. We may want to disentangle illuminant and object colors in color constancy; rendering conditions from surface shape in shape-from-shading; face identity and head pose in face recognition; or font and letter class in character recognition. We refer to these two factors generically as "style" and "content". Bilinear models offer a powerful framework for extracting the two-factor structure of a set of observations, and are familiar in computational vision from several well-known lines of research. This paper shows how bilinear models can be used to learn the style-content structure of a pattern analysis or synthesis problem, which can then be generalized to solve related tasks using different styles and/or content. We focus on three tasks: extrapolating the style of data to unseen content classes, classifying data with known content under a novel style, and translating data from novel content classes and style to a known style or content. We show examples from color constancy, face pose estimation, shape-from-shading, typography and speech.
Model recommendation for action recognition
Authors: Pyry Matikainen, Rahul Sukthankar, Martial Hebert
Venue: Conference CVPR 2012
Year: 2012

Simply choosing one model out of a large set of possibilities for a given vision task is a surprisingly difficult problem, especially if there is limited evaluation data with which to distinguish among models, such as when choosing the best “walk” action classifier from a large pool of classifiers tuned for different viewing angles, lighting conditions, and background clutter. In this paper we suggest that this problem of selecting a good model can be recast as a recommendation problem, where the goal is to recommend a good model for a particular task based on how well a limited probe set of models appears to perform. Through this conceptual remapping, we can bring to bear all the collaborative filtering techniques developed for consumer recommender systems (e.g., Netflix, Amazon.com). We test this hypothesis on action recognition, and find that even when every model has been directly rated on a training set, recommendation finds better selections for the corresponding test set than the best performers on the training set.
Making one object look like another: controlling appearance using a projector-camera system
Authors: Michael D. Grossberg, Harish Peri, Shree K. Nayar, Peter N. Belhumeur
Venue: Conference CVPR 2004
Year: 2004

We present a method for controlling the appearance of an arbitrary 3D object using a projector and a camera. Our goal is to make one object look like another by projecting a carefully determined compensation image onto the object. The determination of the appropriate compensation image requires accounting for spatial variation in the object's reflectance, the effects of environmental lighting, and the spectral responses, spatially varying fall-offs, and non-linear responses in the projector-camera system. Addressing each of these effects, we present a compensation method which calls for the estimation of only a small number of parameters, as part of a novel off-line radiometric calibration. This calibration is accomplished by projecting and acquiring a minimal set of 6 images, irrespective of the object. Results of the calibration are then used on-line to compensate each input image prior to projection. Several experimental results are shown that demonstrate the ability of this method to control the appearance of everyday objects. Our method has direct applications in several areas including smart environments, product design and presentation, adaptive camouflages, interactive education and entertainment.
2d-3d fusion for layer decomposition of urban facades
Authors: Yangyan Li, Qian Zheng, Andrei Sharf, Daniel Cohen-Or, Baoquan Chen, Niloy J. Mitra
Venue: Conference ICCV 2011
Year: 2011

We present a method for fusing two acquisition modes, 2D photographs and 3D LiDAR scans, for depth-layer decomposition of urban facades. The two modes have complementary characteristics: point cloud scans are coherent and inherently 3D, but are often sparse, noisy, and incomplete; photographs, on the other hand, are of high resolution, easy to acquire, and dense, but view-dependent and inherently 2D, lacking critical depth information. In this paper we use photographs to enhance the acquired LiDAR data. Our key observation is that with an initial registration of the 2D and 3D datasets we can decompose the input photographs into rectified depth layers. We decompose the input photographs into rectangular planar fragments and diffuse depth information from the corresponding 3D scan onto the fragments by solving a multi-label assignment problem. Our layer decomposition enables accurate repetition detection in each planar layer, using which we propagate geometry, remove outliers and enhance the 3D scan. Finally, the algorithm produces an enhanced, layered, textured model. We evaluate our algorithm on complex multi-planar building facades, where direct autocorrelation methods for repetition detection fail. We demonstrate how 2D photographs help improve the 3D scans by exploiting data redundancy, and transferring high level structural information to (plausibly) complete large missing regions.
A branch-and-bound algorithm for globally optimal calibration of a camera-and-rotation-sensor system
Authors: Yongduek Seo, Youngju Choi, Sang Wook Lee
Venue: Conference ICCV 2009
Year: 2009

We propose a branch-and-bound algorithm to obtain the globally optimal relative rotation between a camera and the rotation sensor attached to it. Compared to previous methods, our approach directly minimizes the image space error related to the measurements which is very natural for camera-based systems. Our algorithm is based on the observation that we may evaluate the residual when the rotation matrix is known. We propose a feasibility test algorithm for the branch-and-bound to efficiently reduce the search volume of the rotation domain. Experimental results are provided using synthetic and real data sets.
Removing outliers using the l\infty norm
Authors: Kristy Sim, Richard I. Hartley
Venue: Conference CVPR 2006
Year: 2006

Recently, there has been interest in solving geometric vision problems such as triangulation and camera resectioning using L\infty minimization. One key advantage of using the L\infty norm rather than the L2 norm is that the L\infty cost function has a single minimum unlike the commonly used L2 cost function which typically has multiple local minima. However, one drawback of using L\infty minimization is that it is not robust to outliers. By minimizing the L\infty norm instead of the L2 norm, we are, in essence, fitting the outliers and not the good data. Therefore, before one can perform L\infty optimization on a problem, it is first necessary to remove outliers. A popular (but generally unsound) method of removing outliers is to minimize the cost function using standard optimization techniques; then if the residual error is too great, remove the offending measurements and continue. Although this method can fail for simple L2 optimization problems, we show in this paper that for a wide class of L\infty problems it is a valid technique. It is proved that the set of measurements with greatest residual must contain at least one outlier. Thus, if we keep throwing out the measurements with greatest residual, we will eventually remove all outliers in the data. We test this hypothesis on the multiview reconstruction problem and show that even simple strategies for throwing out these maximum residual measurements are effective in removing outliers.
3d face recognition using 3d alignment for pca
Authors: Trina Russ, Chris Boehnen, Tanya Peters
Venue: Conference CVPR 2006
Year: 2006

This paper presents a 3D approach for recognizing faces based on Principal Component Analysis (PCA). The approach addresses the issue of proper 3D face alignment required by PCA for maximum data compression and good generalization performance for new untrained faces. This issue has traditionally been addressed by 2D data normalization, a step that eliminates 3D object size information important for the recognition process. We achieve correspondence of facial points by registering a 3D face to a scaled generic 3D reference face and subsequently perform a surface normal search algorithm. 3D scaling of the generic reference face is performed to enable better alignment of facial points while preserving important 3D size information in the input face. The benefits of this approach for 3D face recognition and dimensionality reduction have been demonstrated on components of the Face Recognition Grand Challenge (FRGC) database versions 1 and 2.
Statistical context priming for object detection
Authors: Antonio Torralba 0001, Pawan Sinha
Venue: Conference ICCV 2001
Year: 2001

There is general consensus that context can be a rich source of information about an object's identity, location and scale. However the issue of how to formalize centextual influences is still largely open. Here we introduce a simple probabilistic framework for modeling the relationship between context and object properties. We represent global context information in terms of the spatial layout of spectral components. The resulting scheme serves as an effective procedure for context driven focus of attention and scale-selection on real-world scenes. Based on a simple holistic analysis of an image, the scheme is able to accurately predict object locations and sizes.
A geometric solver for calibrated stereo egomotion
Authors: Enrique Dunn, Brian Clipp, Jan-Michael Frahm
Venue: Conference ICCV 2011
Year: 2011

This paper introduces a novel geometrical solution for the pose estimation of a stereo camera system as commonly used in robotics, where the camera system balances between coverage and overlap. The proposed approach considers a set of features observed, respectively, in four, three and two views. In contrast to most algebraic solutions our constraints are geometrically meaningful. Initially, we use a four view feature to restrict our translation vector to lie on the surface of a sphere while setting orientation as a function of translation up to a single rotational degree of freedom. Next, we use a three view feature to restrict the translation vector to lie on a circle on the sphere, while completely defining orientation as a function of translation. Finally, we use a two view feature to determine the translation vector lying on the intersection of the circle and one of the generator lines of a doubly ruled quadric. We show how for this final step, the problem can be reduced to the intersection of two coplanar circles. We also analyze the degenerate configurations of the proposed solver and perform an experimental evaluation.
On optimal light configurations in photometric stereo
Authors: Ondrej Drbohlav, Mike J. Chantler
Venue: Conference ICCV 2005
Year: 2005

This paper develops new theory for the optimal placement of photometric stereo lighting in the presence of camera noise. We show that for three lights, any triplet of orthogonal light directions minimises the uncertainty in scaled normal computation. The assumptions are that the camera noise is additive and normally distributed, and uncertainty is defined as the expectation of squared distance of scale normal to the ground truth. If the camera noise is of zero mean and variance sigma <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> the optimal (minimum) uncertainty in the scaled normal is 3sigma <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> For case of n > 3 lights, we show that the minimum uncertainty is 9sigma <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> n, and identify sets of light configurations which reach this theoretical minimum
Robust analysis of feature spaces: Color image segmentation
Authors: Dorin Comaniciu, Peter Meer
Venue: Conference CVPR 1997
Year: 1997

A general technique for the recovery of significant image features is presented. The technique is based on the mean shift algorithm, a simple nonparametric procedure for estimating density gradients. Drawbacks of the current methods (including robust clustering) are avoided. Feature space of any nature can be processed, and as an example, color image segmentation is discussed. The segmentation is completely autonomous, only its class is chosen by the user. Thus, the same program can produce a high quality edge image, or provide, by extracting all the significant colors, a preprocessor for content-based query systems. A 512/spl times/512 color image is analyzed in less than 10 seconds on a standard workstation. Gray level images are handled as color images having only the lightness coordinate.
Finding good composition in panoramic scenes
Authors: Yuan-Yang Chang, Hwann-Tzong Chen
Venue: Conference ICCV 2009
Year: 2009

We introduce a new problem of automatic photo composition, and present an effective technique for finding good views within a panoramic scene. Instead of applying heuristic rules of photo composition, we propose to imitate good composition presented in the artworks of professional photographers. Our approach tries to model the composition styles of professional photographs by analyzing the structural features and the layout of visual saliency. The task of finding good photo composition through a viewfinder is formulated as a search problem, and we present a stochastic search algorithm to look for good viewing configurations and to choose suitable reference images from the collection of masterpiece photographs. Given any initial location in the panoramic scene, our algorithm is able to suggest a better view that would often yield professional-like photo composition.
Statistical modeling and performance characterization of a real-time dual camera surveillance system
Authors: Michael Greiffenhagen, Visvanathan Ramesh, Dorin Comaniciu, Heinrich Niemann
Venue: Conference CVPR 2000
Year: 2000

The engineering of computer vision systems that meet application specific computational and accuracy requirements is crucial to the deployment of real-life computer vision systems. This paper illustrates how past work on a systematic engineering methodology for vision systems performance characterization can be used to develop a real-time people detection and zooming system to meet given application requirements. We illustrate that by judiciously choosing the system modules and performing a careful analysis of the influence of various tuning parameters on the system it is possible to: perform proper statistical inference, automatically set control parameters and quantify limits of a dual-camera real-time video surveillance system. The goal of the system is to continuously provide a high resolution zoomed-in image of a person's head at any location of the monitored area. An omni-directional camera video is processed to detect people and to precisely control a high resolution foveal camera, which has pan, tilt and zoom capabilities. The pan and tilt parameters of the foveal camera and its uncertainties are shown to be functions of the underlying geometry, lighting conditions, background color/contrast, relative position of the person with respect to both cameras as well as sensor noise and calibration errors. The uncertainty in the estimates is used to adaptively estimate the zoom parameter that guarantees with a user specified probability, /spl alpha/, that the detected person's face is contained and zoomed within the image.
A simple and efficient rectification method for general motion
Authors: Marc Pollefeys, Reinhard Koch, Luc Van Gool
Venue: Conference ICCV 1999
Year: 1999

In this paper a new rectification method is proposed. The method is both simple and efficient and can deal with all possible camera motions. A minimal image size without any pixel loss is guaranteed. The only required information is the oriented fundamental matrix. The whole rectification process is carried out directly in the images. The idea consists of using a polar parametrization of the image around the epipole. The transfer between the images is obtained through the fundamental matrix. The proposed method has important advantages compared to the traditional rectification schemes. In some cases these approaches yield very large images or can not rectify at all. Even the recently proposed cylindrical rectification method can encounter problems in some cases. These problems are mainly due to the fact that the matching ambiguity is not reduced to half epipolar lines. Although this last method is more complex than the one proposed in this paper the resulting images are in general larger. The performance of the new approach is illustrated with some results on real image pairs.
Memecraft: Contextual and stance-driven multimodal meme generation
Authors: Han Wang, Roy Ka-Wei Lee
Venue: Conference WWW 2024
Year: 2024

Online memes have emerged as powerful digital cultural artifacts in the age of social media, offering not only humor but also platforms for political discourse, social critique, and information dissemination. Their extensive reach and influence in shaping online communities' sentiments make them invaluable tools for campaigning and promoting ideologies. Despite the development of several meme generation tools, there remains a gap in their systematic evaluation and their ability to effectively communicate ideologies. Addressing this, we introduce MemeCraft, an innovative meme generator that leverages large language models (LLMs) and visual language models (VLMs) to produce memes advocating specific social movements. MemeCraft presents an end-to-end pipeline, transforming user prompts into compelling multimodal memes without manual intervention. Conscious of the misuse potential in creating divisive content, an intrinsic safety mechanism is embedded to curb hateful meme production. Our assessment, focusing on two UN Sustainable Development Goals-Climate Action and Gender Equality-shows MemeCraft's prowess in creating memes that are both funny and supportive of advocacy goals. This paper highlights how generative AI can promote social good and pioneers the use of LLMs and VLMs in meme generation.
Lorentzian residual neural networks
Authors: Neil He, Menglin Yang, Rex Ying
Venue: Conference KDD 2025
Year: 2025

Hyperbolic neural networks have emerged as a powerful tool for modeling hierarchical data structures prevalent in real-world datasets. Notably, residual connections, which facilitate the direct flow of information across layers, have been instrumental in the success of deep neural networks. However, current methods for constructing hyperbolic residual networks suffer from limitations such as increased model complexity, numerical instability, and errors due to multiple mappings to and from the tangent space. To address these limitations, we introduce LResNet, a novel Lorentzian residual neural network based on the weighted Lorentzian centroid in the Lorentz model of hyperbolic geometry. Our method enables the efficient integration of residual connections in Lorentz hyperbolic neural networks while preserving their hierarchical representation capabilities. We demonstrate that our method can theoretically derive previous methods while offering improved stability, efficiency, and effectiveness. Extensive experiments on both graph and vision tasks showcase the superior performance and robustness of our method compared to state-of-the-art Euclidean and hyperbolic alternatives. Our findings highlight the potential of LResNet for building more expressive neural networks in hyperbolic embedding space as a generally applicable method to multiple architectures, including CNNs, GNNs, and graph Transformers.
Multi-task ranking with user behaviors for text-video search
Authors: Peidong Liu, Dongliang Liao, Jinpeng Wang, Yangxin Wu, Gongfu Li, Shu-Tao Xia, Jin Xu
Venue: Conference WWW 2022
Year: 2022

Text-video search has become an important demand in many industrial video sharing platforms, e.g., YouTube, TikTok, and WeChat Channels, thereby attracting increasing research attention. Traditional relevance-based ranking methods for text-video search concentrate on exploiting the semantic relevance between video and query. However, relevance is no longer the principal issue in the ranking stage, because the candidate items retrieved from the matching stage naturally guarantee adequate relevance. Instead, we argue that boosting user satisfaction should be an ultimate goal for ranking and it is promising to excavate cheap and rich user behaviors for model training. To achieve this goal, we propose an effective Multi-Task Ranking pipeline with User Behaviors (MTRUB) for text-video search. Specifically, to exploit the multi-modal data effectively, we put forward a Heterogeneous Multi-modal Fusion Module (HMFM) to fuse the query and video features of different modalities in adaptive ways. Besides that, we design an Independent Multi-modal Input Scheme (IMIS) to alleviate competing task correlation problems in multi-task learning. Experiments on the offline dataset gathered from WeChat Search demonstrate that MTRUB outperforms the baseline by 12.0% in mean gAUC and 13.3% in mean nDCG@10. We also conduct live experiments on a large-scale mobile search engine, i.e., WeChat Search, and MTRUB obtains substantial improvement compared with the traditional relevance-based ranking model.
Homography based multiple camera detection and tracking of people in a dense crowd
Authors: Ran Eshel, Yael Moses
Venue: Conference CVPR 2008
Year: 2008

Tracking people in a dense crowd is a challenging problem for a single camera tracker due to occlusions and extensive motion that make human segmentation difficult. In this paper we suggest a method for simultaneously tracking all the people in a densely crowded scene using a set of cameras with overlapping fields of view. To overcome occlusions, the cameras are placed at a high elevation and only peoplepsilas heads are tracked. Head detection is still difficult since each foreground region may consist of multiple subjects. By combining data from several views, height information is extracted and used for head segmentation. The head tops, which are regarded as 2D patches at various heights, are detected by applying intensity correlation to aligned frames from the different cameras. The detected head tops are then tracked using common assumptions on motion direction and velocity. The method was tested on sequences in indoor and outdoor environments under challenging illumination conditions. It was successful in tracking up to 21 people walking in a small area (2.5 people per m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ), in spite of severe and persistent occlusions.