nils-herrmann/clean_scirepeval huggingface dataset:
Robust optical flow from photometric invariants

Optical flow is widely in use in the field of image processing. In general, optical flow is computed from luminance images. However, optical flow based on luminance information highly depends on moving shadows, varying shading and moving specularities due to camera movement, and fluctuations in the light source intensity. In this paper, we propose a novel method to compute optical flow based on photometric invariants. A major drawback of photometric invariants and their derivatives is that they are unstable for certain RGB values. Therefore, we study on photometric invariant derivatives and noise propagation yielding a condence measure indicating the stability of the corresponding photometric invariant derivatives. This condence measure is integrated into the optical flow framework to provide robustness against noisy data. Experimental results show that the proposed method significantly outperforms optical flow estimation that does not take the instability of the invariants into account.
Per-patch metric learning for robust image matching

We propose a patch-specific metric learning method to improve matching performance of local descriptors. Existing methodologies typically focus on invariance, by completely considering, or completely disregarding all variations. We propose a metric learning method that is robust to only a range of variations. The ability to choose the level of robustness allows us to fine-tune the trade-off between invariance and discriminative power. We learn a distance metric for each patch independently by sampling from a set of relevant image transformations. These transformations give a-priori knowledge about the behavior of the query patch under the applied transformation in feature space. We learn the robust metric by either fully generating only the relevant range of transformations, or by a novel direct metric. The matching between query patch and data is performed with this new metric. Results on the ALOI dataset show that the proposed method improves performance of SIFT by 6.22% for geometric and 4.43% for photometric transformations.
Color Constancy Using 3D Scene Geometry Derived From a Single Image

The aim of color constancy is to remove the effect of the color of the light source. As color constancy is inherently an ill-posed problem, most of the existing color constancy algorithms are based on specific imaging assumptions (e.g., gray-world and white patch assumption). In this paper, 3D geometry models are used to determine which color constancy method to use for the different geometrical regions (depth/layer) found in images. The aim is to classify images into stages (rough 3D geometry models). According to stage models, images are divided into stage regions using hard and soft segmentation. After that, the best color constancy methods are selected for each geometry depth. To this end, we propose a method to combine color constancy algorithms by investigating the relation between depth, local image statistics, and color constancy. Image statistics are then exploited per depth to select the proper color constancy method. Our approach opens the possibility to estimate multiple illuminations by distinguishing nearby light source from distant illuminations. Experiments on state-of-the-art data sets show that the proposed algorithm outperforms state-of-the-art single color constancy algorithms with an improvement of almost 50% of median angular error. When using a perfect classifier (i.e, all of the test images are correctly classified into stages); the performance of the proposed method achieves an improvement of 52% of the median angular error compared with the best-performing single color constancy algorithm.
Reflectance-based classification of color edges

We aim at using color information to classify the physical nature of edges in video. To achieve physics-based edge classification, we first propose a novel approach to color edge detection by automatic noise-adaptive thresholding derived from sensor noise analysis. Then, we present a taxonomy on color edge types. As a result, a parameter-free edge classifier is obtained by labeling color transitions into one of the following types: (1) shadow-geometry, (2) highlight edges, (3) material edges. The proposed method is empirically verified on images showing complex real world scenes.
Learning photometric invariance from diversified color model ensembles

Color is a powerful visual cue for many computer vision applications such as image segmentation and object recognition. However, most of the existing color models depend on the imaging conditions affecting negatively the performance of the task at hand. Often, a reflection model (e.g., Lambertian or dichromatic reflectance) is used to derive color invariant models. However, those reflection models might be too restricted to model real-world scenes in which different reflectance mechanisms may hold simultaneously. Therefore, in this paper, we aim to derive color invariance by learning from color models to obtain diversified color invariant ensembles. First, a photometrical orthogonal and non-redundant color model set is taken on input composed of both color variants and invariants. Then, the proposed method combines and weights these color models to arrive at a diversified color ensemble yielding a proper balance between invariance (repeatability) and discriminative power (distinctiveness). To achieve this, the fusion method uses a multi-view approach to minimize the estimation error. In this way, the method is robust to data uncertainty and produces properly diversified color invariant ensembles. Experiments are conducted on three different image datasets to validate the method. From the theoretical and experimental results, it is concluded that the method is robust against severe variations in imaging conditions. The method is not restricted to a certain reflection model or parameter tuning. Further, the method outperforms state-of- the-art detection techniques in the field of object, skin and road recognition.
Image Alignment by Piecewise Planar Region Matching

Robust image registration is a challenging problem, especially when dealing with severe changes in illumination and viewpoint. Previous methods assume a global geometric model (e.g., homography) and, hence, are only able to align images under predefined constraints (e.g., planar scenes and parallax-free camera motion). However, these constraints may not hold for natural scenes and uncontrolled imaging conditions. Therefore, this paper proposes a novel method which approximates image regions with planes by incorporating piecewise local geometric models. The approximated planar regions are obtained by exploiting a hierarchical figure-ground segmentation method. Each such planar region assumes an affine transformation. To achieve the alignment of the planar regions, an energy function is defined which employs intensity, a key-point descriptor, and geometric information under a global constraint. By re-segmenting and re-merging planar regions iteratively in an energy minimization framework, the method is able to align images even under significant changes in illumination and viewpoint. Experiments on two datasets show that the proposed method outperforms state-of-the-art, especially in the case of large appearance variations and it is, therefore, applicable to web-images (i.e., unconstrained setting) which are taken from the same scene with different viewpoints.
SuperPixel Based Angular Differences as a Mid-level Image Descriptor

This paper focuses on the object recognition task and aims at improving the accuracy with an emphasis on the feature extraction step. Feature extraction is widely used in image classification as an initial step in the pipeline. In this paper, we propose a method to explore the conventional feature extraction techniques from the perspective that mid-level information could be incorporated in order to obtain a superior scene description. We hypothesize that the commonly used pixel based low-level descriptions are useful but can be improved with the introduction of mid-level region information. Hence, we investigate super pixel based image representation to acquire such mid-level information in order to improve the classification accuracy. Detailed experimental evaluations on classification and retrieval tasks are performed in order to validate the proposed hypothesis. A consistent increase is observed in the mean average precision (MAP) score for different experimental scenarios and image categories.
Color mode filtering

In this paper mode filtering of color images is explored. An existing framework based on local histograms is extended to multi-channel images. Within this framework three color mode operations are proposed; 1. Global mode operation for edge sharpening, noise reduction and small object removal, 2. Constrained mode operation for white noise filtering while preserving detail, and 3. Uncertain data mode filtering to incorporate prior knowledge about the certainty of the measurements into the mode computation. Results obtained for a variety of images indicate the feasibility of color mode filtering.
Boosting saliency in color image features

The aim of salient point detection is to find distinctive events in images. Salient features are generally determined from the local differential structure of images. They focus on the shape saliency of the local neighborhood. The majority of these detectors is luminance based which has the disadvantage that the distinctiveness of the local color information is completely ignored. To fully exploit the possibilities of color image salient point detection, color distinctiveness should be taken into account next to shape distinctiveness. In this paper color distinctiveness is explicitly incorporated into the design of saliency detection. The algorithm, called color saliency boosting, is based on an analysis of the statistics of color image derivatives. Isosalient color derivatives can be closely approximated by ellipsoidal surfaces in color derivative space. Based on this remarkable statistical finding, isosalient derivatives are transformed by color boosting to have equal impact on the saliency. Color saliency boosting is designed as a generic method easily adaptable to existing feature detectors. Results show that substantial improvements in information content are acquired by targeting color salient features. Further, the generality of the method is illustrated by applying color boosting to multiple existing saliency methods.
Extracting 3D Layout From a Single Image Using Global Image Structures

Extracting the pixel-level 3D layout from a single image is important for different applications, such as object localization, image, and video categorization. Traditionally, the 3D layout is derived by solving a pixel-level classification problem. However, the image-level 3D structure can be very beneficial for extracting pixel-level 3D layout since it implies the way how pixels in the image are organized. In this paper, we propose an approach that first predicts the global image structure, and then we use the global structure for fine-grained pixel-level 3D layout extraction. In particular, image features are extracted based on multiple layout templates. We then learn a discriminative model for classifying the global layout at the image-level. Using latent variables, we implicitly model the sublevel semantics of the image, which enrich the expressiveness of our model. After the image-level structure is obtained, it is used as the prior knowledge to infer pixel-wise 3D layout. Experiments show that the results of our model outperform the state-of-the-art methods by 11.7% for 3D structure classification. Moreover, we show that employing the 3D structure prior information yields accurate 3D scene layout segmentation.
Like Father, Like Son: Facial Expression Dynamics for Kinship Verification

Kinship verification from facial appearance is a difficult problem. This paper explores the possibility of employing facial expression dynamics in this problem. By using features that describe facial dynamics and spatio-temporal appearance over smile expressions, we show that it is possible to improve the state of the art in this problem, and verify that it is indeed possible to recognize kinship by resemblance of facial expressions. The proposed method is tested on different kin relationships. On the average, 72.89% verification accuracy is achieved on spontaneous smiles.
Classifying color transitions into shadow-geometry, illumination, highlight or material edges

We aim at using color information to classify the physical nature of a color edge: that is whether the transition is due to shadows, abrupt surface orientation changes, illumination, highlights or material changes. To achieve a physics-based edge classification, we propose a taxonomy of color invariant edges. The taxonomy is based upon the sensitivity of the various color edges with respect to different imaging dependencies i.e. shadows, object shape, shading (i.e. illumination intensity changes), highlights and material characteristics. From this taxonomy, the edge classifier is derived labeling color transitions into the following types: (1) shadow, geometry or shading edges, (2) highlight edges, (3) material edges. Experiments conducted with the edge classification technique on color and hyperspectral images show that the proposed method successfully discriminates the different edge types.
Geometry-constrained spatial pyramid adaptation for image classification

This paper proposes a geometry-constrained spatial pyramid adaptation approach for the image classification task. Scene geometry is used as an input parameter for generating the spatial pyramid definitions. The resulting region adaptation is performed in accordance with the predefined geometric guidelines and underlying image characteristics. Using an approximate global geometric correspondence, exploits the idea that images of the same category share a spatial similarity. This assumption is evaluated and justified in an object classification framework, in which generated region segments are used as an enhancement to the widely utilized “spatial pyramid” method. Fixed region pyramids are replaced by the proposed locally coherent geometrically consistent region segments. Performance of the proposed method on object classification framework is evaluated on the 20 class Pascal VOC 2007 dataset. The proposed method shows consistent increase in the mean average precision (MAP) score for different experimental scenarios.
Con-Text: Text Detection for Fine-Grained Object Classification

This paper focuses on fine-grained object classification using recognized scene text in natural images. While the state-of-the-art relies on visual cues only, this paper is the first work which proposes to combine textual and visual cues. Another novelty is the textual cue extraction. Unlike the state-of-the-art text detection methods, we focus more on the background instead of text regions. Once text regions are detected, they are further processed by two methods to perform text recognition, i.e., ABBYY commercial OCR engine and a state-of-the-art character recognition algorithm. Then, to perform textual cue encoding, bi- and trigrams are formed between the recognized characters by considering the proposed spatial pairwise constraints. Finally, extracted visual and textual cues are combined for fine-grained classification. The proposed method is validated on four publicly available data sets: ICDAR03, ICDAR13, Con-Text, and Flickr-logo. We improve the state-of-the-art end-to-end character recognition by a large margin of 15% on ICDAR03. We show that textual cues are useful in addition to visual cues for fine-grained classification. We show that textual cues are also useful for logo retrieval. Adding textual cues outperforms visual- and textual-only in fine-grained classification (70.7% to 60.3%) and logo retrieval (57.4% to 54.8%).
Color feature detection and classification by learning

In this paper, we aim at the classification of the physical nature of local image structures in color images on the basis of geometrical and photometrical information. To this end, a framework is proposed to combine the local differential structure (i.e. geometrical information such as edges, corners, T-junctions etc) and color (i.e. photometrical information such as shadows, shading, illumination, highlights) in a multi-dimensional feature space. This framework is used to yield a proper classifier to classify salient image structures on the basis of their physical nature. The proposed framework is empirically verified on a set of images. From the theoretical and experimental results it is concluded that the proposed classification scheme successfully classifies local image structures robust to image translation and rotation, illumination intensity variations, and noise.
Robust segmentation and tracking of colored objects in video

Segmenting and tracking of objects in video is of great importance for video-based encoding, surveillance, and retrieval. However, the inherent difficulty of object segmentation and tracking is to distinguish changes in the displacement of objects from disturbing effects such as noise and illumination changes. Therefore, in this paper, we formulate a color-based deformable model which is robust against noisy data and changing illumination. Computational methods are presented to measure color constant gradients. Further, a model is given to estimate the amount of sensor noise through these color constant gradients. The obtained uncertainty is subsequently used as a weighting term in the deformation process. Experiments are conducted on image sequences recorded from three-dimensional scenes. From the experimental results, it is shown that the proposed color constant deformable method successfully finds object contours robust against illumination, and noisy, but homogeneous regions.
Image segmentation and similarity of color-texture objects

We aim for content-based image retrieval of textured objects in natural scenes under varying illumination and viewing conditions. To achieve this, image retrieval is based on matching feature distributions derived from color invariant gradients. To cope with object cluttering, region-based texture segmentation is applied on the target images prior to the actual image retrieval process. The retrieval scheme is empirically verified on color images taken from textured objects under different lighting conditions.
Computational Color Constancy: Survey and Experiments

Computational color constancy is a fundamental prerequisite for many computer vision applications. This paper presents a survey of many recent developments and state-of-the-art methods. Several criteria are proposed that are used to assess the approaches. A taxonomy of existing algorithms is proposed and methods are separated in three groups: static methods, gamut-based methods, and learning-based methods. Further, the experimental setup is discussed including an overview of publicly available datasets. Finally, various freely available methods, of which some are considered to be state of the art, are evaluated on two datasets.
Sigma nigma: an image retrieval system

Presents a system which retrieves images on the basis of automatically generated indexes (i.e. semantic image representations, obtained by automatic image analysis, indicating the content of the images). The system consists of two parts: an off-line indexing part and an on-line image retrieval part. The indexing component is used to automatically generate semantic representations of images so that the image retrieval component can use this information to enable image retrieval. The man-machine communication of the image retrieval component is based on an iconical graphical query language to accomplish geographical query specification for image access. Experiments have been carried out on three different sets of images from the following domains: MRI images of the chest, electronic schemas and topographic maps. The experiments show encouraging results especially for domains which have a high degree of formality in their pictorial expression, such as electronic schemas and topographic maps, and to a less extent for domains having a weak degree of formality such as MRI images of the chest.<<ETX>>
User assisted stereo image retargetting

The wide availability of stereoscopic 3D TVs and mobile displays created a considerable market for content producers. This opened a new phase for researchers to study on methods to alter and process the content for various purposes. This study concentrates on user assisted disparity remapping for stereo image footage. The disparity of an object of interest is altered while leaving the remaining scene unattended. This application is useful in the sense that it provides a method for emphasizing/de-emphasizing an object on the scene by adjusting its depth with respect to the camera. The proposed technique can also be used as a post processing step for retargeting stereoscopic footage on different display sizes and resolutions. The proposed technique involves an MRF based energy minimization step for interactive stereo image segmentation, where user assistance on just one of the stereo pairs is required for determining the location of stereo object pair. Qualitative and quantitative evaluation of the generated segments and the disparity adjusted images prove the functionality and superiority of the proposed technique.
Recognition of Genuine Smiles

Automatic distinction between genuine (spontaneous) and posed expressions is important for visual analysis of social signals. In this paper, we describe an informative set of features for the analysis of face dynamics, and propose a completely automatic system to distinguish between genuine and posed enjoyment smiles. Our system incorporates facial landmarking and tracking, through which features are extracted to describe the dynamics of eyelid, cheek, and lip corner movements. By fusing features over different regions, as well as over different temporal phases of a smile, we obtain a very accurate smile classifier. We systematically investigate age and gender effects, and establish that age-specific classification significantly improves the results, even when the age is automatically estimated. We evaluate our system on the 400-subject UvA-NEMO database we have recently collected, as well as on three other smile databases from the literature . Through an extensive experimental evaluation, we show that our system improves the state of the art in smile classification and provides useful insights in smile psychophysics.