nils-herrmann/clean_scirepeval huggingface dataset:
Fast Construction of Covariance Matrices for Arbitrary Size Image Windows

We propose an integral image based algorithm to extract feature covariance matrices of all possible rectangular regions within a given image. Covariance is an essential indicator of how much the deviation of two or more variables match. In our case, these variables correspond to point-wise features, e.g. coordinates, color values, gradients, edge magnitude and orientation, local histograms, filter responses, etc. We significantly improve the speed of the covariance computation by taking advantage of the spatial arrangement of image points using integral images, which are intermediate representations used for calculation of region sums. Each point of the integral image corresponds to the summation of all point values inside the feature image rectangle bounded by the upper left corner and the point of interest. Using this representation, any rectangular region sum can be computed in constant time. We follow a similar idea for fast calculation of region covariance. We construct integral images for all separate features as well as integral images of the multiplication of any two feature combinations. Using these set of integral images and region corner point coordinates, we directly extract the covariance matrix coefficients. We show that the proposed method reduces the computational load to quadratic time.
Automatic threshold determination of centroid-linkage region growing by MPEG-7 dominant color descriptors

The emerging MPEG-7 standard embodies a visual descriptor that is associated with the dominant colors of an image. A threshold adaptation method for region based image and video segmentation that takes advantage of the MPEG-7 dominant color descriptor is presented. This method enables assignment of region growing parameters without any low-level processing. In the standard, it is proposed that the dominant colors be extracted by clustering of color histograms. This property is used to determine color homogeneity that is formulated into the Lorentzian-based color distance norm and corresponding thresholds. The proposed algorithm is compared with other region growing algorithms, and results show that the threshold adaptation performs faster and more robustly.
Probabilistic Visual Tracking via Robust Template Matching and Incremental Subspace Update

In this paper, we present a probabilistic algorithm for visual tracking that incorporates robust template matching and incremental sub-space update. There are two template matching methods used in the tracker: one is robust to small perturbation and the other to background clutter. Each method yields a probability of matching. Further, the templates are modeled using mixed probabilities and updated once the templates in the library cannot capture the variation of object appearance. We also model the tracking history using a nonlinear subspace that is described by probabilistic kernel principal components analysis, which provides a third probability. The most-recent tracking result is added to the nonlinear subspace incrementally. This update is performed efficiently by augmenting the kernel Gram matrix with one row and one column. The product of the three probabilities is defined as the observation likelihood used in a particle filter to derive the tracking result. Experimental results demonstrate the efficiency and effectiveness of the proposed algorithm.
Color alignment in texture mapping of images under point light source and general lighting condition

Recently, creating digital contents through measurements of the real world has become an important technique in the field of virtual reality, digital preservation of cultural heritages and so forth. One of the effective methods for creating such digital contents is texture mapping, which maps color images on a 3D geometric model. This method has two problems: geometric and photometric. The geometric problem lies mostly in how to estimate the camera position and rotation relative to the 3D position of the target object. The photometric problem deals with color consistencies between input textures. Most studies on texture mapping focus on the geometric problem, while only a few works deal with the photometric problem. In this paper we propose two novel methods for color alignment in texture mapping. One method is based on the fact that chromaticity of images is independent from their geometric data. The other method utilizes the basis image of a point light source on an illumination sphere. In both methods, the pseudo-albedo is estimated and the color consistency between the input images is accomplished.
Depth Map Completion by Jointly Exploiting Blurry Color Images and Sparse Depth Maps

We aim at predicting a complete and high-resolution depth map from incomplete, sparse and noisy depth measurements. Existing methods handle this problem either by exploiting various regularizations on the depth maps directly or resorting to learning based methods. When the corresponding color images are available, the correlation between the depth maps and the color images are used to improve the completion performance, assuming the color images are clean and sharp. However, in real world dynamic scenes, color images are often blurry due to the camera motion and the moving objects in the scene. In this paper, we propose to tackle the problem of depth map completion by jointly exploiting the blurry color image sequences and the sparse depth map measurements, and present an energy minimization based formulation to simultaneously complete the depth maps, estimate the scene flow and deblur the color images. Our experimental evaluations on both outdoor and indoor scenarios demonstrate the state-of-the-art performance of our approach.
Joint tracking and video registration by factorial Hidden Markov models

Tracking moving objects from image sequences obtained by a moving camera is a difficult problem since there exists apparent motion of the static background. It becomes more difficult when the camera motion between the consecutive frames is very large. Traditionally, registration is applied before tracking to compensate for the camera motion using parametric motion models. At the same time, the tracking result highly depends on the performance of registration. This raises problems when there are big moving objects in the scene and the registration algorithm is prone to fail, since the tracker easily drifts away when poor registration results occur. In this paper, we tackle this problem by registering the frames and tracking the moving objects simultaneously within the factorial hidden Markov model framework using particle filters. Under this framework, tracking and registration are not working separately, but mutually benefit each other by interacting. Particles are drawn to provide the candidate geometric transformation parameters and moving object parameters.  according to the geometric transformation parameters by maximizing a joint gradient function. A state-of-the-art covariance tracker is used to track the moving object. The tracking score is obtained by incorporating both background and foreground information. By using knowledge of the position of the moving objects, we avoid blindly registering the image pairs without taking the moving object regions into account. We apply our algorithm to moving object tracking on numerous image sequences with camera motion and show the robustness and effectiveness of our method.
Scene Categorization with Spectral Features

Spectral signatures of natural scenes were earlier found to be distinctive for different scene types with varying spatial envelope properties such as openness, naturalness, ruggedness, and symmetry. Recently, such handcrafted features have been outclassed by deep learning based representations. This paper proposes a novel spectral description of convolution features, implemented efficiently as a unitary transformation within deep network architectures. To the best of our knowledge, this is the first attempt to use deep learning based spectral features explicitly for image classification task. We show that the spectral transformation decorrelates convolutional activations, which reduces co-adaptation between feature detections, thus acts as an effective regularizer. Our approach achieves significant improvements on three large-scale scene-centric datasets (MIT-67, SUN-397, and Places-205). Furthermore, we evaluated the proposed approach on the attribute detection task where its superior performance manifests its relevance to semantically meaningful characteristics of natural scenes.
A new method for tracking performance evaluation based on a reflective model and perturbation analysis

In this paper, we present a novel methodology for tracking performance evaluation. Considering the continuity of the image sequences in a video, we define a new measurement called tracking difficulty which incorporates the local sequence information among a small image sequence centered at each frame. We subsequently use a reflective model to formulate tracking difficulty. Tracking difficulty curves can not only illustrate at which parts of the video one tracking algorithm performs well or poor, but also provide a way to compare the performance of different tracking algorithms. We further add perturbation analysis to the reflective model to examine how sensitive the tracking algorithm is to noise. Results on data sets are presented to show the effectiveness of our evaluation method.
Connecting the dots in multi-class classification: From nearest subspace to collaborative representation

We present a novel multi-class classifier that strikes a balance between the nearest-subspace classifier, which assigns a test sample to the class that minimizes the distance between the test sample and its principal projection in the selected class, and a collaborative representation based classifier, which classifies a sample to the class that minimizes the distance between the collaborative components of the test sample by using all training samples from all classes as the dictionary and its projection in the selected class. In our formulation, the sparse representation based classifier  and nearest subspace classifier become special cases under different regularization parameters. We show that the classification performance can be improved by optimally tuning the regularization parameter, which can be done at almost no extra computational cost. We give extensive numerical examples for digit identification and face recognition with performance comparisons of different choices of collaborative representations, in particular when only a partial observation of the test sample is available via compressive sensing measurements.
Joint Discriminative Bayesian Dictionary and Classifier Learning

We propose to jointly learn a Discriminative Bayesian dictionary along a linear classifier using coupled Beta-Bernoulli Processes. Our representation model uses separate base measures for the dictionary and the classifier, but associates them to the class-specific training data using the same Bernoulli distributions. The Bernoulli distributions control the frequency with which the factors (e.g. dictionary atoms) are used in data representations, and they are inferred while accounting for the class labels in our approach. To further encourage discrimination in the dictionary, our model uses separate (sets of) Bernoulli distributions to represent data from different classes. Our approach adaptively learns the association between the dictionary atoms and the class labels while tailoring the classifier to this relation with a joint inference over the dictionary and the classifier. Once a test sample is represented over the dictionary, its representation is accurately labelled by the classifier due to the strong coupling between the dictionary and the classifier. We derive the Gibbs Sampling equations for our joint representation model and test our approach for face, object, scene and action recognition to establish its effectiveness.
Category-Specific Object Image Denoising

We present a novel image denoising algorithm that uses external, category specific image database. In contrast to existing noisy image restoration algorithms that search patches either from a generic database or noisy image itself, our method first selects clean images similar to the noisy image from a database that consists of images of the same class. Then, within the spatial locality of each noisy patch, it assembles a set of “support patches” from the selected images. These noisy-free support samples resemble the noisy patch and correspond principally to the identical part of the depicted object. In addition, we employ a content adaptive distribution model for each patch, where we derive the parameters of the distribution from the support patches. We formulate noise removal task as an optimization problem in the transform domain. Our objective function composed of a Gaussian fidelity term that imposes category specific information, and a low-rank term that encourages the similarity between the noisy and the support patches in a robust manner. The denoising process is driven by an iterative selection of support patches and optimization of the objective function. Our extensive experiments on five different object categories confirm the benefit of incorporating category-specific information to noise removal and demonstrate the superior performance of our method over the state-of-the-art alternatives.
Deep Salient Object Detection by Integrating Multi-level Cues

A key problem in salient object detection is how to effectively exploit the multi-level saliency cues in a unified and data-driven manner. In this paper, building upon the recent success of deep neural networks, we propose a fully convolutional neural network based approach empowered with multi-level fusion to salient object detection. By integrating saliency cues at different levels through fully convolutional neural networks and multi-level fusion, our approach could effectively exploit both learned semantic cues and higher-order region statistics for edge-accurate salient object detection. First, we fine-tune a fully convolutional neural network for semantic segmentation to adapt it to salient object detection to learn a suitable yet coarse perpixel saliency prediction map. This map is often smeared across salient object boundaries since the local receptive fields in the convolutional network apply naturally on both sides of such boundaries. Second, to enhance the resolution of the learned saliency prediction and to incorporate higher-order cues that are omitted by the neural network, we propose a multi-level fusion approach where super-pixel level coherency in saliency is exploited. Our extensive experimental results on various benchmark datasets demonstrate that the proposed method outperforms the state-of the-art approaches.
Imagining the Unimaginable Faces by Deconvolutional Networks

We tackle the challenge of constructing 64 pixels for each individual pixel of a thumbnail face image. We show that such an aggressive super-resolution objective can be attained by taking advantage of the global context and making the best use of the prior information portrayed by the image class. Our input image is so small (e.g.,  $16\times 16$  pixels) that it can be considered as a patch of itself. Thus, conventional patch-matching-based super-resolution solutions are unsuitable. In order to enhance the resolution while enforcing the global context, we incorporate a pixel-wise appearance similarity objective into a deconvolutional neural network, which allows efficient learning of mappings between low-resolution input images and their high-resolution counterparts in the training data set. Furthermore, the deconvolutional network blends the learned high-resolution constituent parts in an authentic manner, where the face structure is naturally imposed and the global context is preserved. To account for the possible artifacts in upsampled feature maps, we employ a sub-network composed of additional convolutional layers. During training, we use roughly aligned images (only eye locations), yet demonstrate that our network has the capacity to super-resolve face images regardless of pose and facial expression variations. This significantly reduces the requirement of precisely face alignments in the data set. Owing to the network topology we apply, our method is robust to translational misalignments. In addition, our method is able to upsample rotational unaligned faces with data augmentation. Our extensive experimental analysis manifests that our method achieves more appealing and superior results than the state of the art.
More about VLAD: A leap from Euclidean to Riemannian manifolds

This paper takes a step forward in image and video coding by extending the well-known Vector of Locally Aggregated Descriptors (VLAD) onto an extensive space of curved Riemannian manifolds. We provide a comprehensive mathematical framework that formulates the aggregation problem of such manifold data into an elegant solution. In particular, we consider structured descriptors from visual data, namely Region Covariance Descriptors and linear subspaces that reside on the manifold of Symmetric Positive Definite matrices and the Grassmannian manifolds, respectively. Through rigorous experimental validation, we demonstrate the superior performance of this novel Riemannian VLAD descriptor on several visual classification tasks including video-based face recognition, dynamic scene recognition, and head pose classification.
Kernel methods for weakly supervised mean shift clustering

Mean shift clustering is a powerful unsupervised data analysis technique which does not require prior knowledge of the number of clusters, and does not constrain the shape of the clusters. The data association criteria is based on the underlying probability distribution of the data points which is defined in advance via the employed distance metric. In many problem domains, the initially designed distance metric fails to resolve the ambiguities in the clustering process. We present a novel semi-supervised kernel mean shift algorithm where the inherent structure of the data points is learned with a few user supplied constraints in addition to the original metric. The constraints we consider are the pairs of points that should be clustered together. The data points are implicitly mapped to a higher dimensional space induced by the kernel function where the constraints can be effectively enforced. The mode seeking is then performed on the embedded space and the approach preserves all the advantages of the original mean shift algorithm. Experiments on challenging synthetic and real data clearly demonstrate that significant improvements in clustering accuracy can be achieved by employing only a few constraints.
Integrated deep and shallow networks for salient object detection

Deep convolutional neural network (CNN) based salient object detection methods have achieved state-of-the-art performance and outperform those unsupervised methods with a wide margin. In this paper, we propose to integrate deep and unsupervised saliency for salient object detection under a unified framework. Specifically, our method takes results of unsupervised saliency (Robust Background Detection, RBD) and normalized color images as inputs, and directly learns an end-to-end mapping between inputs and the corresponding saliency maps. The color images are fed into a Fully Convolutional Neural Networks (FCNN) adapted from semantic segmentation to exploit high-level semantic cues for salient object detection. Then the results from deep FCNN and RBD are concatenated to feed into a shallow network to map the concatenated feature maps to saliency maps. Finally, to obtain a spatially consistent saliency map with sharp object boundaries, we fuse superpixel level saliency map at multi-scale. Extensive experimental results on 8 benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art approaches with a margin.
Inter-camera color calibration by correlation model function

A novel solution to the inter-camera color calibration problem, which is very important for multicamera systems is presented. We propose a distance metric and a model function to evaluate the inter-camera radiometric properties. Instead of depending on the shape assumptions of brightness transfer function to find separate radiometric responses, we derive a nonparametric function to model color distortion for pair-wise camera combinations. Our method is based on correlation matrix analysis and dynamic programming. The correlation matrix is computed from three 1-D color histograms, and the model function is obtained from a minimum cost path traced within the matrix. The model function enables accurate compensation of color mismatches, which cannot be done with conventional distance metrics. Furthermore, we show that our metric can be reduced to other commonly used metrics with suitable simplification. Our simulations prove the effectiveness of the proposed method even for severe color distortions.
Constant time O(1) bilateral filtering

This paper presents three novel methods that enable bilateral filtering in constant time O(1) without sampling. Constant time means that the computation time of the filtering remains same even if the filter size becomes very large. Our first method takes advantage of the integral histograms to avoid the redundant operations for bilateral filters with box spatial and arbitrary range kernels. For bilateral filters constructed by polynomial range and arbitrary spatial filters, our second method provides a direct formulation by using linear filters of image powers without any approximation. Lastly, we show that Gaussian range and arbitrary spatial bilateral filters can be expressed by Taylor series as linear filter decompositions without any noticeable degradation of filter response. All these methods drastically decrease the computation time by cutting it down constant times (e.g. to 0.06 seconds per 1MB image) while achieving very high PSNRpsilas over 45 dB. In addition to the computational advantages, our methods are straightforward to implement.
Simultaneous Stereo Video Deblurring and Scene Flow Estimation

Videos for outdoor scene often show unpleasant blur effects due to the large relative motion between the camera and the dynamic objects and large depth variations. Existing works typically focus monocular video deblurring. In this paper, we propose a novel approach to deblurring from stereo videos. In particular, we exploit the piece-wise planar assumption about the scene and leverage the scene flow information to deblur the image. Unlike the existing approach  which used a pre-computed scene flow, we propose a single framework to jointly estimate the scene flow and deblur the image, where the motion cues from scene flow estimation and blur information could reinforce each other, and produce superior results than the conventional scene flow estimation or stereo deblurring methods. We evaluate our method extensively on two available datasets and achieve significant improvement in flow estimation and removing the blur effect over the state-of-the-art methods.