nils-herrmann/clean_scirepeval huggingface dataset:
Empowering Visual Categorization With the GPU

Visual categorization is important to manage large collections of digital images and video, where textual metadata is often incomplete or simply unavailable. The bag-of-words model has become the most powerful method for visual categorization of images and video. Despite its high accuracy, a severe drawback of this model is its high computational cost. As the trend to increase computational power in newer CPU and GPU architectures is to increase their level of parallelism, exploiting this parallelism becomes an important direction to handle the computational cost of the bag-of-words approach. When optimizing a system based on the bag-of-words approach, the goal is to minimize the time it takes to process batches of images. this paper, we analyze the bag-of-words model for visual categorization in terms of computational cost and identify two major bottlenecks: the quantization step and the classification step. We address these two bottlenecks by proposing two efficient algorithms for quantization and classification by exploiting the GPU hardware and the CUDA parallel programming model. The algorithms are designed to (1) keep categorization accuracy intact, (2) decompose the problem, and (3) give the same numerical results. In the experiments on large scale datasets, it is shown that, by using a parallel implementation on the Geforce GTX260 GPU, classifying unseen images is 4.8 times faster than a quad-core CPU version on the Core i7 920, while giving the exact same numerical results. In addition, we show how the algorithms can be generalized to other applications, such as text retrieval and video retrieval. Moreover, when the obtained speedup is used to process extra video frames in a video retrieval benchmark, the accuracy of visual categorization is improved by 29%.
3D Scene priors for road detection

Vision–based road detection is important in different areas of computer vision such as autonomous driving, car collision warning and pedestrian crossing detection. However, current vision–based road detection methods are usually based on low–level features and they assume structured roads, road homogeneity, and uniform lighting conditions. Therefore, in this paper, contextual 3D information is used in addition to low–level cues. Low–level photometric invariant cues are derived from the appearance of roads. Contextual cues used include horizon lines, vanishing points, 3D scene layout and 3D road stages. Moreover, temporal road cues are included. All these cues are sensitive to different imaging conditions and hence are considered as weak cues. Therefore, they are combined to improve the overall performance of the algorithm. To this end, the low-level, contextual and temporal cues are combined in a Bayesian framework to classify road sequences. Large scale experiments on road sequences show that the road detection method is robust to varying imaging conditions, road types, and scenarios (tunnels, urban and highway). Further, using the combined cues outperforms all other individual cues. Finally, the proposed method provides highest road detection accuracy when compared to state–of–the–art methods.
DENSE sampling of features for image retrieval

This paper focuses on the image retrieval task. We propose the use of dense feature points computed on several color channels to improve the retrieval system. To validate our approach, an evaluation of various SIFT extraction strategies is performed. Detected SIFT are compared with dense SIFT. Dense color descriptors: C-SIFT and T-SIFT are then utilized. A comparison between standard and rotation invariant features is further achieved. Finally, several encoding strategies are studied: Bag of Visual Words (BOW), Fisher vectors, and vector of locally aggregated descriptors (VLAD). The presented approaches are evaluated on several datasets and we show a large improvement over the baseline.
Segmentation as selective search for object recognition

For object recognition, the current state-of-the-art is based on exhaustive search. However, to enable the use of more expensive features and classifiers and thereby progress beyond the state-of-the-art, a selective search strategy is needed. Therefore, we adapt segmentation as a selective search by reconsidering segmentation: We propose to generate many approximate locations over few and precise object delineations because (1) an object whose location is never generated can not be recognised and (2) appearance and immediate nearby context are most effective for object recognition. Our method is class-independent and is shown to cover 96.7% of all objects in the Pascal VOC 2007 test set using only 1,536 locations per image. Our selective search enables the use of the more expensive bag-of-words method which we use to substantially improve the state-of-the-art by up to 8.5% for 8 out of 20 classes on the Pascal VOC 2010 detection challenge.
Automatic landmarking and alignment for facial expression analysis

In this paper we develop a fully automatic emotional expression recognition system. We propose improvements on a recent facial landmark localization method. This method is then used to bootstrap a Bezie´r volume tracker for tracking facial features in video. The motion of the facial features is used in classifying the expression of the face. We present extensive cross-database results for the landmarking and expression recognition modules, and show that automatic landmarking significantly improves the subsequent expression recognition.
Selection and Fusion of Color Models for Image Feature Detection

The choice of a color model is of great importance for many computer vision algorithms (e.g., feature detection, object recognition, and tracking) as the chosen color model induces the equivalence classes to the actual algorithms. As there are many color models available, the inherent difficulty is how to automatically select a single color model or, alternatively, a weighted subset of color models producing the best result for a particular task. The subsequent hurdle is how to obtain a proper fusion scheme for the algorithms so that the results are combined in an optimal setting. To achieve proper color model selection and fusion of feature detection algorithms, in this paper, we propose a method that exploits nonperfect correlation between color models or feature detection algorithms derived from the principles of diversification. As a consequence, a proper balance is obtained between repeatability and distinctiveness. The result is a weighting scheme which yields maximal feature discrimination. The method is verified experimentally for three different image feature detectors. The experimental results show that the fusion method provides feature detection results having a higher discriminative power than the standard weighting scheme. Further, it is experimentally shown that the color model selection scheme provides a proper balance between color invariance (repeatability) and discriminative power (distinctiveness)
Robustifying Descriptor Instability Using Fisher Vectors

Many computer vision applications, including image classification, matching, and retrieval use global image representations, such as the Fisher vector, to encode a set of local image patches. To describe these patches, many local descriptors have been designed to be robust against lighting changes and noise. However, local image descriptors are unstable when the underlying image signal is low. Such low-signal patches are sensitive to small image perturbations, which might come e.g., from camera noise or lighting effects. In this paper, we first quantify the relation between the signal strength of a patch and the instability of that patch, and second, we extend the standard Fisher vector framework to explicitly take the descriptor instabilities into account. In comparison to common approaches to dealing with descriptor instabilities, our results show that modeling local descriptor instability is beneficial for object matching, image retrieval, and classification.
Color constancy based on the Grey-edge hypothesis

A well-known color constancy method is based on the Grey-World assumption i.e. the average reflectance of surfaces in the world is achromatic. In this article we propose a new hypothesis for color constancy, namely the Grey-Edge hypothesis assuming that the average edge difference in a scene is achromatic. Based on this hypothesis, we propose an algorithm for color constancy. Recently, the Grey-World hypothesis and the max-RGB method were shown to be two instantiations of a Minkowski norm based color constancy method. Similarly we also propose a more general version of the Grey-Edge hypothesis which assumes that the Minkowsky norm of derivatives of the reflectance of surfaces is achromatic. The algorithms are tested on a large data set of images under different illuminants, and the results show that the new method outperforms the Grey-World assumption and the max-RGB method. Results are comparable to more elaborate algorithms, however at lower computational costs.
Image segmentation by directed region subdivision

In this paper, an image segmentation method based on directed image region partitioning is proposed. The method consists of two separate stages: a splitting phase followed by a merging phase. The splitting phase starts with an initial coarse triangulation and employs the incremental Delaunay triangulation as a directed image region splitting technique. The triangulation process is accomplished by adding points as vertices one by one into the triangulation. A top-down point selection strategy is proposed for selecting these points in the image domain of grey-value and color images. The merging phase coalesces the oversegmentation, generated by the splitting phase, into homogeneous image regions. Because images might be negatively affected by changes in intensity due to shading or surface orientation change, the authors propose homogeneity criteria which are robust to intensity changes caused by these phenomena for both grey-value and color images. Performance of the image segmentation method has been evaluated by experiments on test images.
Evaluating Color Descriptors for Object and Scene Recognition

Image category recognition is important to access visual information on the level of objects and scene types. So far, intensity-based descriptors have been widely used for feature extraction at salient points. To increase illumination invariance and discriminative power, color descriptors have been proposed. Because many different descriptors exist, a structured overview is required of color invariant descriptors in the context of image category recognition. Therefore, this paper studies the invariance properties and the distinctiveness of color descriptors (software to compute the color descriptors from this paper is available from http://www.colordescriptors.com) in a structured way. The analytical invariance properties of color descriptors are explored, using a taxonomy based on invariance properties with respect to photometric transformations, and tested experimentally using a data set with known illumination conditions. In addition, the distinctiveness of color descriptors is assessed experimentally using two benchmarks, one from the image domain and one from the video domain. From the theoretical and experimental results, it can be derived that invariance to light intensity changes and light color changes affects category recognition. The results further reveal that, for light intensity shifts, the usefulness of invariance is category-specific. Overall, when choosing a single descriptor and no prior knowledge about the data set and object and scene categories is available, the OpponentSIFT is recommended. Furthermore, a combined set of color descriptors outperforms intensity-based SIFT and improves category recognition by 8 percent on the PASCAL VOC 2007 and by 7 percent on the Mediamill Challenge.
Edge and corner detection by photometric quasi-invariants

Feature detection is used in many computer vision applications such as image segmentation, object recognition, and image retrieval. For these applications, robustness with respect to shadows, shading, and specularities is desired. Features based on derivatives of photometric invariants, which we is called full invariants, provide the desired robustness. However, because computation of photometric invariants involves nonlinear transformations, these features are unstable and, therefore, impractical for many applications. We propose a new class of derivatives which we refer to as quasi-invariants. These quasi-invariants are derivatives which share with full photometric invariants the property that they are insensitive for certain photometric edges, such as shadows or specular edges, but without the inherent instabilities of full photometric invariants. Experiments show that the quasi-invariant derivatives are less sensitive to noise and introduce less edge displacement than full invariant derivatives. Moreover, quasi-invariants significantly outperform the full invariant derivatives in terms of discriminative power.
Skin detection using the EM algorithm with spatial constraints

In this paper, we propose a color-based method for skin detection and segmentation, which also takes into account the spatial coherence of the skin pixels. We treat the problem of skin detection as an inference problem. We assume that each pixel in an image has a hidden binary label associated with it, that specifies if it is skin or not. In order to solve the inference problem, we use a variational EM algorithm, which incorporates the spatial constraints with just a small computational overhead in the E-step. Finally, we show that our method provides better results than the standard EM algorithm and a state-of-art skin-detection method from the literature of Jones, M. J. and Rehg, J. M. (2002).
Color Constancy using Natural Image Statistics

Although many color constancy methods exist, they are all based on specific assumptions such as the set of possible light sources, or the spatial and spectral characteristics of images. As a consequence, no algorithm can be considered as universal. However, with the large variety of available methods, the question is how to select the method that induces equivalent classes for different image characteristics. Furthermore, the subsequent question is how to combine the different algorithms in a proper way. To achieve selection and combining of color constancy algorithms, in this paper, natural image statistics are used to identify the most important characteristics of color images. Then, based on these image characteristics, the proper color constancy algorithm (or best combination of algorithms) is selected for a specific image. To capture the image characteristics, the Weibull parameterization (e.g. texture and contrast) is used. Experiments show that, on a large data set of 11,000 images, our approach outperforms current state-of-the-art single algorithms, as well as simple alternatives for combining several algorithms.
Image indexing using composite color and shape invariant features

New sets of color models are proposed for object recognition invariant to a change in view point, object geometry and illumination. Further, computational methods are presented to combine color and shape invariants to produce a high-dimensional invariant feature set for discriminatory object recognition. Experiments on a database of 500 images show that object recognition based on composite color and shape invariant features provides excellent recognition accuracy. Furthermore, object recognition based on color invariants provides very high recognition accuracy whereas object recognition based entirely on shape invariants yields very poor discriminative power. The image database and the performance of the recognition scheme can be experienced within PicToSeek: on-line as part of the ZOMAX system at: http://www.wins.uva.nl/research/isis/zomax/.
Estimation of Sunlight Direction Using 3D Object Models

The direction of sunlight is an important informative cue in a number of applications in image processing, such as augmented reality and object recognition. In general, existing methods to estimate the direction of the sunlight rely on different image features (e.g., sky, texture, shadows, and shading). These features can be considered as weak informative cues as no single feature can reliably estimate the sunlight direction. Moreover, existing methods may require that the camera parameters are known limiting their applicability. In this paper, we present a new method to estimate the sunlight direction from a single (outdoor) image by inferring casts shadows through object modeling and recognition. First, objects (e.g., cars or persons) are first (automatically) recognized in images by exemplar-SVMs. Instead of training the Support Vector Machine (SVMs) using natural images (limited variation in viewpoints), we propose to train on 2D object samples generated from 3D object models. Then, the recognized objects are used as sundial cues (probes) to estimate the sunlight direction by inferring the corresponding shadows generated by 3D object models considering different illumination directions. We demonstrate the effectiveness of our approach on synthetic and real images. Experiments show that our method estimates the azimuth angle accurately within a quadrant (smaller than 45°) and compute the zenith angle with mean angular error of 23°.
Classifying multimedia documents by merging textual and pictorial information

In this paper, we study computational models and techniques to merge textual and image features to classify multimedia documents into semantically meaningful groups. A vector-based framework is used to index documents on the basis of textual, pictorial and composite (textual-pictorial) information. The scheme makes use of weighted document terms and color invariant image features to obtain a high-dimensional image descriptor in vector form to be used as an index. Based on supervised learning, a classifier is used to organize the multimedia documents. Due to space limitations, in this paper, we focus on the application of classifying/finding pictures of people on the Internet. Performance evaluations are reported on the accuracy of merging textual and pictorial information for classification.
Evaluation of Color Spatio-Temporal Interest Points for Human Action Recognition

This paper considers the recognition of realistic human actions in videos based on spatio-temporal interest points (STIPs). Existing STIP-based action recognition approaches operate on intensity representations of the image data. Because of this, these approaches are sensitive to disturbing photometric phenomena, such as shadows and highlights. In addition, valuable information is neglected by discarding chromaticity from the photometric representation. These issues are addressed by color STIPs. Color STIPs are multichannel reformulations of STIP detectors and descriptors, for which we consider a number of chromatic and invariant representations derived from the opponent color space. Color STIPs are shown to outperform their intensity-based counterparts on the challenging UCF sports, UCF11 and UCF50 action recognition benchmarks by more than 5% on average, where most of the gain is due to the multichannel descriptors. In addition, the results show that color STIPs are currently the single best low-level feature choice for STIP-based approaches to human action recognition.
Geographic information for vision-based road detection

Road detection is a vital task for the development of autonomous vehicles. The knowledge of the free road surface ahead of the target vehicle can be used for autonomous driving, road departure warning, as well as to support advanced driver assistance systems like vehicle or pedestrian detection. Using vision to detect the road has several advantages in front of other sensors: richness of features, easy integration, low cost or low power consumption. Common vision-based road detection approaches use low-level features (such as color or texture) as visual cues to group pixels exhibiting similar properties. However, it is difficult to foresee a perfect clustering algorithm since roads are in outdoor scenarios being imaged from a mobile platform. In this paper, we propose a novel high-level approach to vision-based road detection based on geographical information. The key idea of the algorithm is exploiting geographical information to provide a rough detection of the road. Then, this segmentation is refined at low-level using color information to provide the final result. The results presented show the validity of our approach.
Point Light Source Position Estimation From RGB-D Images by Learning Surface Attributes

Light source position (LSP) estimation is a difficult yet an important problem in computer vision. A common approach for estimating the LSP assumes Lambert’s law. However, in real-world scenes, Lambert’s law does not hold for all different types of surfaces. Instead of assuming all that surfaces follow Lambert’s law, our approach classifies image surface segments based on their photometric and geometric surface attributes (i.e. glossy, matte, curved, and so on) and assigns weights to image surface segments based on their suitability for LSP estimation. In addition, we propose the use of the estimated camera pose to globally constrain LSP for RGB-D video sequences. Experiments on Boom and a newly collected RGB-D video data sets show that the state-of-the-art methods are outperformed by the proposed method. The results demonstrate that weighting image surface segments based on their attributes outperform the state-of-the-art methods in which the image surface segments are considered to equally contribute. In particular, by using the proposed surface weighting, the angular error for LSP estimation is reduced from 12.6° to 8.2° and 24.6° to 4.8° for Boom and RGB-D video data sets, respectively. Moreover, using the camera pose to globally constrain LSP provides higher accuracy (4.8°) compared with using single frames (8.5°).
Shadow edge detection using geometric and photometric features

The detection of shadow and shading edges is a first step towards reducing the imaging effects that are caused by interactions of the light source with surfaces that are in the scene. As most of the algorithms for shadow edge detection use photometric information, geometric information have been ignored so far. In this paper, the aim is to include geometric features for more robust shadow edge detection. First, thousands of patches are annotated as either containing a shadow edge or not. Then, geometric features of these patches are analyzed and it is shown that the combination of photometric and geometric features improves the classification of shadow edges with respect to using either one of these features with 14%. These results demonstrate the added value of geometric features, in addition to photometric features, for the detection of shadow edges.
Robust histogram construction from color invariants

A simple and effective object recognition scheme is to represent and march images on the basis of color histograms. To obtain robustness against varying imaging circumstances (e.g. a change in illumination, object pose, and viewpoint), color histograms are constructed from color invariants. However in general, color invariants are negatively affected by sensor noise due to the instabilities of these color invariant transforms at many RGB values. To suppress the effect of noise blow-up for unstable color invariant values, in this paper color invariant histograms are computed using variable kernel density estimators. To apply variable kernel density estimation in a principled way, models are proposed for the propagation of sensor noise through color invariants. As a result the associated uncertainty is known for each color invariant value. The associated uncertainty is used to derive the parameterization of the variable kernel density estimator during histogram construction. It is empirically verified that the proposed method compares favorably to traditional color histograms for object recognition.