nils-herrmann/clean_scirepeval huggingface dataset:
Progressive 3D Modeling All the Way

This work proposes a method bridging the existing gap between progressive sparse 3D reconstruction (incremental Structure from Motion) and progressive point based dense 3D reconstruction (Multi-View Stereo). The presented algorithm is capable of adapting an existing dense 3D model to changes such as the addition or removal of new images, the merge of scene parts, or changes in the underlying camera calibration. The existing 3D model is transformed as consistently as possible and the structure is reused as much as possible without sacrificing the accuracy and/or completeness of the final result. A significant decrease in runtime is achieved compared to the re-computation of a new dense point cloud from scratch. We demonstrate the performance of the algorithm in various experiments on publicly available datasets of different sizes and compare it to the baseline. The work interacts seamlessly with publicly available software enabling an integrated progressive 3D modeling pipeline.
A Hough transform-based voting framework for action recognition

We present a method to classify and localize human actions in video using a Hough transform voting framework. Random trees are trained to learn a mapping between densely-sampled feature patches and their corresponding votes in a spatio-temporal-action Hough space. The leaves of the trees form a discriminative multi-class codebook that share features between the action classes and vote for action centers in a probabilistic manner. Using low-level features such as gradients and optical flow, we demonstrate that Hough-voting can achieve state-of-the-art performance on several datasets covering a wide range of action-recognition scenarios.
Multi-label image segmentation via point-wise repetition

Bottom-up segmentation tends to rely on local features. Yet, many natural and man-made objects contain repeating elements. Such structural and more spread-out features are important cues for segmentation but are more difficult to exploit. The difficulty also comes from the fact that repetition need not be perfect, and will actually rather be partial, approximate, or both in most cases. This paper presents a multi-label image segmentation algorithm that processes a single input image and efficiently discovers and exploits repeating elements without any prior knowledge about their shape, color or structure. The algorithm spells out the interplay between segmentation and repetition detection. The key of our approach is a novel, point-wise concept of repetition. This is defined by point-wise mutual information and locally compares certain neighborhoods to accumulate evidence. This point-wise repetition measure naturally handles imperfect repetitions, and the parts with inconsistent appearances are recognized and assigned with low scores. An energy functional is proposed to include the point-wise repetition into the image segmentation process, which takes the form of a graph-cut minimization. Real scene images demonstrate the ability of our algorithm to handle partial and approximate repetition.
Elastic convolved ICP for the registration of deformable objects

This paper describes a method for registering deformable 3D objects. When an object such as a hand deforms, the deformation of the local shape is small, whereas the global shape deforms to a greater extent in many cases. Therefore, the local shape can be used as a feature for matching corresponding points. Instead of using a descriptor of the local shape, we introduce the convolution of the error between corresponding points for each vertex of a 3D mesh model. This approach is analogous to window matching in 2D image registration. Since the proposed method computes the convolution for every vertex in a model, it incorporates dense feature matching as opposed to sparse matching based on certain feature descriptors. Through experiments, we show that the convolution is useful for finding corresponding points and evaluate the accuracy of the registration.
Real-time accurate stereo with bitwise fast voting on CUDA

This paper proposes a real-time design for accurate stereo matching on Compute Unified Device Architecture (CUDA). We adopt a leading local algorithm for its high data parallelism. A GPU-oriented bitwise fast voting method is proposed to effectively improve the matching accuracy, which is enormously faster than the histogram-based approach. The whole algorithm is parallelized on CUDA at a fine granularity, efficiently exploiting the computing resources of GPUs. On-chip shared memory is utilized to alleviate the latency of memory accesses. Compared to the CPU counterpart, our design attains a speedup factor of 52. With high matching accuracy, the proposed design is still among the fastest stereo methods on GPUs. The advantages of speed and accuracy advocate our design for practical applications such as robotics systems and multiview teleconferencing.
Mobile phone and cloud — A dream team for 3D reconstruction

Recently, Structure-from-Motion pipelines (SfM) for the 3D reconstruction of scenes from images were pushed from desktop computers onto mobile devices, like phones or tablets. However, mobile devices offer much more than just necessary computational power. A combination of handheld device with camera, display and full connectivity entails possibilities for an on-line 3D reconstruction that would have been difficult to implement otherwise. In this work, we propose a combination of a regular mobile phone as frontend with a centralized server plus annex cloud as backend for collaborative, on-line 3D reconstruction. We illustrate few advantages of this combination of a myriad of new possibilities: First, we automatically balance computational load between the frontend and the backend depending on battery autonomy and available bandwidth. Second, we select the best of algorithms given the available resources to obtain better 3D models. Finally, we allow for collaborative modeling in order to arrive at more complete and more detailed models, especially when the objects or scenes are big. This paper presents an implementation of such a joint mobile-cloud modeling approach and demonstrates its advantages via real-life reconstructions.
Make my day - high-fidelity color denoising with Near-Infrared

We address the task of restoring RGB images taken under low illumination (e.g. night time), when an aligned near infrared (NIR or simply N) image taken under stronger NIR illumination is available. Such restoration holds the promise that algorithms designed to work under daylight conditions could be used around the clock. Increasingly, RGBN cameras are becoming available, as car cameras tend to include a Near-Infrared (N) band, next to R, G, and B bands, and NIR artificial lighting is applied. Under low lighting conditions, the NIR band is less noisy than the others and this is all the more the case if stronger illumination is only available in the NIR band. We address the task of restoring the R, G, and B bands on the basis of the NIR band in such cases. Even if the NIR band is less strongly correlated with the R, G, and B bands than these bands are mutually, there is sufficient such correlation to pick up important textural and gradient information in the NIR band and inject it into the others. The algorithm that we propose - coined `Make My Day' or MMD for short - is akin to the previously published BM3D denoising algorithm. MMD denoises the three (visible - NIR) differential images to then add back the original NIR image. It not only effectively reduces the noise but also includes the texture and edge information in the high spatial frequency range. MMD outperforms other state-of-art denoising methods in terms of PSNR, texture quality, and color fidelity. We publish our codes and images.
Multi-view manhole detection, recognition, and 3D localisation

Our work addresses the problem of accurately 3D localising specific types of road fixtures, such as manhole covers. The surveying task for manholes has to be done for millions of kilometres of road. We propose an efficient pipeline that starts from images captured by vans, which are then used to detect, recognise and localise the manholes. Challenges come from frequent occlusions, regular changes in illumination conditions, substantial viewpoint variance, and the fact that the actual manhole covers are far less similar than one might expect. We combine 2D and 3D techniques and demonstrate high performance for large amounts of image data. To our knowledge this is the first published work for manhole mapping based solely on computer vision techniques and GPS.
Systematic evaluation of super-resolution using classification

Currently two evaluation methods of super-resolution (SR) techniques prevail: The objective Peak Signal to Noise Ratio (PSNR) and a qualitative measure based on manual visual inspection. Both of these methods are sub-optimal: The latter does not scale well to large numbers of images, while the former does not necessarily reflect the perceived visual quality. We address these issues in this paper and propose an evaluation method based on image classification. We show that perceptual image quality measures like structural similarity are not suitable for evaluation of SR methods. On the other hand a systematic evaluation using large datasets of thousands of real-world images provides a consistent comparison of SR algorithms that corresponds to perceived visual quality. We verify the success of our approach by presenting an evaluation of three recent super-resolution algorithms on standard image classification datasets.
WEAR++: 3D model driven camera tracking on board the International Space Station

We present WEAR++, a wearable augmented reality system consisting of a head mounted display, a camera and an inertial measurement unit. This paper focuses on the visual camera tracking system developed for WEAR++. Using a 3D model of the scene, we first create a map of 3D-2D correspondences in an off line mapping procedure. During on line operation, we match features from a new image to the database, and track the camera pose with an Extended Kalman Filter using the recovered 3D-2D correspondences. By using robust local features (SURF) and a frustum culling algorithm, we demonstrate that we are able to track the pose even for jerky motions and blurry images. Furthermore, we explain how the system was utilised by astronaut Frank De Winne on board the International Space Station for performing maintenance tasks.
Superpixel meshes for fast edge-preserving surface reconstruction

Multi-View-Stereo (MVS) methods aim for the highest detail possible, however, such detail is often not required. In this work, we propose a novel surface reconstruction method based on image edges, superpixels and second-order smoothness constraints, producing meshes comparable to classic MVS surfaces in quality but orders of magnitudes faster. Our method performs per-view dense depth optimization directly over sparse 3D Ground Control Points (GCPs), hence, removing the need for view pairing, image rectification, and stereo depth estimation, and allowing for full per-image parallelization. We use Structure-from-Motion (SfM) points as GCPs, but the method is not specific to these, e.g. LiDAR or RGB-D can also be used. The resulting meshes are compact and inherently edge-aligned with image gradients, enabling good-quality lightweight per-face flat renderings. Our experiments demonstrate on a variety of 3D datasets the superiority in speed and competitive surface quality.
The Interestingness of Images

We investigate human interest in photos. Based on our own and others' psychophysical experiments, we identify various cues for "interestingness", namely aesthetics, unusualness and general preferences. For the ranking of retrieved images, interestingness shows to be more appropriate than cues proposed earlier. Interestingness is correlated with what people believe they will remember. This is opposed to actual memorability, which is uncorrelated to both. We introduce a set of features computationally capturing the three main aspects of visual interestingness and build an interestingness predictor from them. Its performance is shown on three datasets with varying context, reflecting the prior knowledge of the viewers.
Reconstructing 3D dynamic scenes with non-accidentalness principle

Relative scale ambiguity is one of the most important problems we encounter while reconstructing dynamic scenes from monocular image sequences. As an example, consider a scene where a car is moving on the road and a moving camera is recording it. By just looking at the images, we can not decide whether it is a real car on the road or a small toy car hovering in front of the camera. In this paper, we first analyze this problem and then present our solutions which are based on the non-accidentalness principle.
Tracking the invisible: Learning where the object might be

Objects are usually embedded into context. Visual context has been successfully used in object detection tasks, however, it is often ignored in object tracking. We propose a method to learn supporters which are, be it only temporally, useful for determining the position of the object of interest. Our approach exploits the General Hough Transform strategy. It couples the supporters with the target and naturally distinguishes between strongly and weakly coupled motions. By this, the position of an object can be estimated even when it is not seen directly (e.g., fully occluded or outside of the image region) or when it changes its appearance quickly and significantly. Experiments show substantial improvements in model-free tracking as well as in the tracking of “virtual” points, e.g., in medical applications.
Visual Recognition in RGB Images and Videos by Learning from RGB-D Data

In this work, we propose a framework for recognizing RGB images or videos by learning from RGB-D training data that contains additional depth information. We formulate this task as a new unsupervised domain adaptation (UDA) problem, in which we aim to take advantage of the additional depth features in the source domain and also cope with the data distribution mismatch between the source and target domains. To handle the domain distribution mismatch, we propose to learn an optimal projection matrix to map the samples from both domains into a common subspace such that the domain distribution mismatch can be reduced. Such projection matrix can be effectively optimized by exploiting different strategies. Moreover, we also use different ways to utilize the additional depth features. To simultaneously cope with the above two issues, we formulate a unified learning framework called domain adaptation from multi-view to single-view (DAM2S). By defining various forms of regularizers in our DAM2S framework, different strategies can be readily incorporated to learn robust SVM classifiers for classifying the target samples, and three methods are developed under our DAM2S framework. We conduct comprehensive experiments for object recognition, cross-dataset and cross-view action recognition, which demonstrate the effectiveness of our proposed methods for recognizing RGB images and videos by learning from RGB-D data.
Regressor Basis Learning for anchored super-resolution

A+ aka Adjusted Anchored Neighborhood Regression - is a state-of-the-art method for exemplar-based single image super-resolution with low time complexity at both train and test time. By robustly training a clustered regression model over a low-resolution dictionary, its performance keeps improving with the dictionary size - even when using tens of thousands of regressors. However, this can pose a memory issue where the model size can grow to more than a gigabyte, limiting applicability in memory constrained scenarios. To address this, we propose Regressor Basis Learning (RB), a novel variant of A+ where we restrict the regressor set to a learned low-dimensional subspace, such that each regressor is coded as a linear combination of few basis regressors. We learn the regressor basis by alternating between closed form solutions of the optimal coding of the regressor set (given the basis) and the optimal regressor basis (given the coding). We validate RB on several standard benchmarks and achieve comparable performance to A+ but by using orders of magnitude fewer basis regressors, ie. 32 basis regressors instead of 1024 regressors. This makes our RB method ideal for memory constrained applications.
Anchored Regression Networks Applied to Age Estimation and Super Resolution

We propose the Anchored Regression Network (ARN), a nonlinear regression network which can be seamlessly integrated into various networks or can be used stand-alone when the features have already been fixed. Our ARN is a smoothed relaxation of a piecewise linear regressor through the combination of multiple linear regressors over soft assignments to anchor points. When the anchor points are fixed the optimal ARN regressors can be obtained with a closed form global solution, otherwise ARN admits end-toend learning with standard gradient based methods. We demonstrate the power of the ARN by applying it to two very diverse and challenging tasks: age prediction from face images and image super-resolution. In both cases, ARNs yield strong results.
Real-time range scanning of deformable surfaces by adaptively coded structured light

We present a new active range scanning technique suitable for moving or deformable surfaces. It is a 'one-shot' system, in that 3D data are acquired from a single image. The projection pattern consists of equidistant black and white stripes combined with a limited number of colored, transversal stripes which aid in their identification. Instead of using a generic, static code that is supposed to work under all circumstances, a 'self-adaptive code' is used. Two modes of adaptation are provided. The first mode is off-line, and generates a robust identification code for the current projector-camera configuration. This configuration is supposed to remain fixed during the remainder of the 3D capturing session. The second mode occurs on-line. By introducing a feedback-loop from the reconstruction output to the pattern generation, the pattern can be adapted as to keep its decoding well-conditioned. Within the considered family of patterns, its parameters are optimized on the fly based on the current content of the scene. As a matter of fact, scene content of the current frame influences the pattern projected during the next. This poses a weak temporal continuity constraint. Only a very fast change of scene invalidates this assumption. The higher the throughput of the reconstruction pipeline, the less serious this constraint becomes. Our current pipeline is running at approximately 20 Hz. Our prototype only uses 'off the shelf' hardware.
Discovery of Sets of Mutually Orthogonal Vanishing Points in Videos

While vanishing point (VP) estimation has received extensive attention, most approaches focus on static images or perform detection and tracking separately. In this paper, we focus on man-made environments and propose a novel method for detecting and tracking groups of mutually orthogonal vanishing points (MOVP), also known as Manhattan frames, jointly from monocular videos. The method is unique in that it is designed to enforce orthogonality in groups of VPs, temporal consistency of each individual MOVP, and orientation consistency of all putative MOVP. To this end, the method consists of three steps: 1) proposal of MOVP candidates by directly incorporating mutual orthogonality; 2) extracting consistent tracks of MOVPs by minimizing the flow cost over a network where nodes are putative MOVPs and edges are putative links across time; and 3) refinement of all MOVPs by enforcing consistency between lines, their identified vanishing directions and consistency of global camera orientation. The method is evaluated on six newly collected and annotated videos of urban scenes. Extensive experiments show that the method outperforms greedy MOVP tracking method considerably. In addition, we also test the method for camera orientation estimation and show that it obtains very promising results on a challenging street-view dataset.
Active MAP Inference in CRFs for Efficient Semantic Segmentation

Most MAP inference algorithms for CRFs optimize an energy function knowing all the potentials. In this paper, we focus on CRFs where the computational cost of instantiating the potentials is orders of magnitude higher than MAP inference. This is often the case in semantic image segmentation, where most potentials are instantiated by slow classifiers fed with costly features. We introduce Active MAP inference 1) to on-the-fly select a subset of potentials to be instantiated in the energy function, leaving the rest of the parameters of the potentials unknown, and 2) to estimate the MAP labeling from such incomplete energy function. Results for semantic segmentation benchmarks, namely PASCAL VOC 2010 and MSRC-21, show that Active MAP inference achieves similar levels of accuracy but with major efficiency gains.
Efficient regression priors for post-processing demosaiced images

Color demosaicing is a process of reconstructing lost pixels in an incomplete color image. By extracting spatial-spectral correlations of RGB channels various interpolation methods have been proposed with low computational complexity. Meanwhile, optimization strategies such as sparsity and adaptive PCA based algorithm (SAPCA) were developed. SAPCA outperforms many interpolation techniques by impressive margins at the cost of dramatically increasing the computational time. In this paper we propose an efficient novel post-processing algorithm based on the adjusted anchored neighborhood regression (A+) method from image super-resolution literature. We greatly improve the results of the demosaicing methods, and achieve image quality as competitive as SAPCA but orders of magnitude faster.
Lococo: low complexity corner detector

The high-speed feature detection is still very demanding for many computer vision applications. In this paper, the Harris and KLT corner detectors are redesigned to reduce the complexity of the algorithm. The complexity of Harris and KLT corner detectors are reduced by using the box kernel, the integral image and efficient non-maximum suppression, achieving complexity reduction by a factor of 8. For the image of size 1000×700, our method costs only 74ms on a commodity CPU with 2GHz and 1GB RAM.
Deep Temporal Linear Encoding Networks

The CNN-encoding of features from entire videos for the representation of human actions has rarely been addressed. Instead, CNN work has focused on approaches to fuse spatial and temporal networks, but these were typically limited to processing shorter sequences. We present a new video representation, called temporal linear encoding (TLE) and embedded inside of CNNs as a new layer, which captures the appearance and motion throughout entire videos. It encodes this aggregated information into a robust video feature representation, via end-to-end learning. Advantages of TLEs are: (a) they encode the entire video into a compact feature representation, learning the semantics and a discriminative feature space, (b) they are applicable to all kinds of networks like 2D and 3D CNNs for video classification, and (c) they model feature interactions in a more expressive way and without loss of information. We conduct experiments on two challenging human action datasets: HMDB51 and UCF101. The experiments show that TLE outperforms current state-of-the-art methods on both datasets.