nils-herrmann/clean_scirepeval huggingface dataset:
Contextually Customized Video Summaries Via Natural Language

The best summary of a long video differs among different people due to its highly subjective nature. Even for the same person, the best summary may change with time or mood. In this paper, we introduce the task of generating contextually customized video summaries through simple text. First, we train a deep architecture to effectively learn semantic embeddings of video frames by leveraging the abundance of image-caption data via a progressive manner, whereby our algorithm is able to select semantically relevant video segments for a contextually meaningful video summary, given a user-specific text description or even a single sentence. In order to evaluate our customized video summaries, we conduct experimental comparison with baseline methods that utilize ground-truth information. Despite the challenging baselines, our method still manages to show comparable or even exceeding performance. We also demonstrate that our method is able to automatically generate semantically diverse video summaries even without any text input.
Fusing Multiple Independent Estimates via Spectral Clustering for Robust Visual Tracking

One fundamental problem of object tracking is the convergence of estimates to local maxima not corresponding to target objects. To mitigate this problem, constructing a good posterior distribution of the target state is important. In this letter, we propose a robust tracking approach by building a new posterior distribution model from multiple independent estimates of a target state. For each candidate of the target state, we compute a confidence score based on its spatial consistency with other estimates and photometric similarities with target models. Our posterior distribution model reflects tracking uncertainties well and adaptively defines the search region for the next frame. We validate the robustness of our approach on a number of challenging datasets.
Reflection removal using disparity and gradient-sparsity via smoothing algorithm

The purpose of this paper is to introduce a new method for removing reflections from multi-view images taken through a transparent medium, such as pane glass. Our method utilizes an optimization approach based on the probabilistic model of relative smoothness algorithm, which exploits gradient value to separate the image into two sub-layers. As this algorithm has certain limitations on removing reflections, we improve upon it by imposing a gradient-sparsity constraint. This allows the type of reflection captured within a camera's focal length to be effectively removed. We also introduce another constraint on a disparity map which smooths specific areas of reflection layer while simultaneously preserves the sharpness of the main object. These two major contributions are proven to be sufficient in producing high-quality images. Our algorithm demonstrates good results compared with other existing methods as most of the reflection spots have been removed, and the computational time of our system is arguably fast.
Complementation of cameras and lasers for accurate 6D SLAM: From correspondences to bundle adjustment

In this paper, we present an accurate and robust 6D SLAM method that uses multiple 2D sensors, i.e. perspective cameras and planar laser scanners. We have investigated strengths and weaknesses of those two sensors for 6D SLAM by conducting specifically designed experiments, and found that the sensors can complement each other. In order to take full advantages of each approach, we fuse correspondences of those two sensors, rather than individually estimated motions. Correspondences obtained by the two sensors have different characteristics, but can be expressed in a common 2D-3D relation form. We use the correspondences in a single structure-from-motion framework. In the initial motion estimation step, we propose a RANSAC-based method to generate and test multiple motion hypotheses by using multiple pools of correspondences, aiming to avoid potential bias of each sensor data. In the later motion refinement step, we introduce a variant of bundle adjustment to consider different types of constraints from the two sensors. The performance of the proposed method is demonstrated both quantitatively by experiments on closed-loop sequences and qualitatively by large-scale experiments with DGPS trajectory. The proposed method successfully closes a loop of 320 meters in twenty thousand frames by incremental process only.
A Tensor-Based Algorithm for High-Order Graph Matching

This paper addresses the problem of establishing correspondences between two sets of visual features using higher order constraints instead of the unary or pairwise ones used in classical methods. Concretely, the corresponding hypergraph matching problem is formulated as the maximization of a multilinear objective function over all permutations of the features. This function is defined by a tensor representing the affinity between feature tuples. It is maximized using a generalization of spectral techniques where a relaxed problem is first solved by a multidimensional power method and the solution is then projected onto the closest assignment matrix. The proposed approach has been implemented, and it is compared to state-of-the-art algorithms on both synthetic and real data.
Architecture of behavior-based mobile robot in dynamic environment

The authors are developing a compact cart-type robot system which moves around an office. They propose a behavior-based architecture with three clustered (reflexive, purposive, and adaptive) agents that realizes efficiency in attaining the mission of the robot, and robustness against the various kinds of failures that may occur in a dynamic environment. The reflexive-level group consists of agents with contact, infrared, and ultrasonic sensors which maintain minimal safety of the robot. The role of the purposive-level group is to achieve the global mission of the robot, such as 'if the robot detects a small fire in the office, find and reach a fire extinguisher as soon as possible'. The adaptive-level group stands by to recover from failure in the purposive-level group or in a deadlock situation. Experimental results showed the effectiveness of the method.<<ETX>>
Efficient and Robust Color Consistency for Community Photo Collections

We present an efficient technique to optimize color consistency of a collection of images depicting a common scene. Our method first recovers sparse pixel correspondences in the input images and stacks them into a matrix with many missing entries. We show that this matrix satisfies a rank two constraint under a simple color correction model. These parameters can be viewed as pseudo white balance and gamma correction parameters for each input image. We present a robust low-rank matrix factorization method to estimate the unknown parameters of this model. Using them, we improve color consistency of the input images or perform color transfer with any input image as the source. Our approach is insensitive to outliers in the pixel correspondences thereby precluding the need for complex pre-processing steps. We demonstrate high quality color consistency results on large photo collections of popular tourist landmarks and personal photo collections containing images of people.
Pixel-Level Matching for Video Object Segmentation Using Convolutional Neural Networks

We propose a novel video object segmentation algorithm based on pixel-level matching using Convolutional Neural Networks (CNN). Our network aims to distinguish the target area from the background on the basis of the pixel-level similarity between two object units. The proposed network represents a target object using features from different depth layers in order to take advantage of both the spatial details and the category-level semantic information. Furthermore, we propose a feature compression technique that drastically reduces the memory requirements while maintaining the capability of feature representation. Two-stage training (pretraining and fine-tuning) allows our network to handle any target object regardless of its category (even if the object’s type does not belong to the pre-training data) or of variations in its appearance through a video sequence. Experiments on large datasets demonstrate the effectiveness of our model - against related methods - in terms of accuracy, speed, and stability. Finally, we introduce the transferability of our network to different domains, such as the infrared data domain.
Robust Computer Vision Techniques for High-Quality 3D Modeling

In this paper, we present our recent sensor fusion approaches to obtain high-quality 3D information. We first discuss two fusion methods that combine geometric and photometric information. The first method, multiview photometric stereo, reconstructs the full 3D shape of a target object. The geometric and photometric information is efficiently fused by using a planar mesh representation. The second method performing shape-from shading with a Kinect sensor estimates the shape of an object under uncalibrated natural illumination. Since the method uses a single RGB-D input, it is capable of capturing the high quality shape details of a dynamic object under varying illumination. Subsequently, we summarize a calibration algorithm of a time of-flight (ToF) sensor and a camera fusion system with a 2.5D pattern. Lastly, we present a camera-laser sensor fusion system for the large-scale 3D reconstruction.
Locally adaptive support-weight approach for visual correspondence search

In this paper, we present a new area-based method for visual correspondence search that focuses on the dissimilarity computation. Local and area-based matching methods generally measure the similarity (or dissimilarity) between the image pixels using local support windows. In this approach, an appropriate support window should be selected adaptively for each pixel to make the measure reliable and certain. Finding the optimal support window with an arbitrary shape and size is, however, very difficult and generally known as an NP-hard problem. For this reason, unlike the existing methods that try to find an optimal support window, we adjusted the support-weight of each pixel in a given support window. The adaptive support-weight of a pixel is computed based on the photometric and geometric relationship with the pixel under consideration. Dissimilarity is then computed using the raw matching costs and support-weights of both support windows, and the correspondence is finally selected by the WTA (winner-takes-all) method. The experimental results for the rectified real images show that the proposed method successfully produces piecewise smooth disparity maps while preserving sharp depth discontinuities accurately.
Probabilistic matching of lines for their homography

A probabilistic matching of lines, which form a homography in two images, is formulated in the framework of the forward stepwise regression. A membership matrix represents the likelihood of line correspondences to the homography. The correspondence measure is borrowed from the forward stepwise regression so that the squared error of the homography and the number of correspondences are balanced simultaneously. An alternating scheme for optimizing the membership and homography is provided. The experimental results on synthetic and real images validate the proposed method.
Time-of-Flight Sensor Calibration for a Color and Depth Camera Pair

We present a calibration method of a time-of-flight (ToF) sensor and a color camera pair to align the 3D measurements with the color image correctly. We have designed a 2.5D pattern board with irregularly placed holes to be accurately detected from low resolution depth images of a ToF camera as well as from high resolution color images. In order to improve the accuracy of the 3D measurements of a ToF camera, we propose to perform ray correction and range bias correction. We reset the transformation of the ToF sensor which transforms the radial distance into the scene depth in Cartesian coordinate through ray correction. Then we capture a planar scene from different depths to correct the distance error that is shown to be dependent not only on the distance but also on the pixel location. The range error profiles along the calibrated distance are classified according to their wiggling shapes and each cluster of profiles with similar shape are separately estimated using a B-spline function. The standard deviation of the remaining random noise is recorded as an uncertainty information of distance measurements. We show the performance of our calibration method quantitatively and qualitatively on various datasets, and validate the impact of our method by demonstrating an RGB-D shape refinement application.
A tensor-based algorithm for high-order graph matching

This paper addresses the problem of establishing correspondences between two sets of visual features using higher-order constraints instead of the unary or pairwise ones used in classical methods. Concretely, the corresponding hypergraph matching problem is formulated as the maximization of a multilinear objective function over all permutations of the features. This function is defined by a tensor representing the affinity between feature tuples. It is maximized using a generalization of spectral techniques where a relaxed problem is first solved by a multi-dimensional power method, and the solution is then projected onto the closest assignment matrix. The proposed approach has been implemented, and it is compared to state-of-the-art algorithms on both synthetic and real data.
Large object detection in cluttered background using boosted Markov Chain Monte Carlo

In this paper, we present a new object detection method using codebook and boosted Markov Chain Monte Carlo (MCMC) estimation. It is relatively well detected using adaboost and simple Haar-like features for small objects. However, the detection problem is more difficult when object size becomes larger (over 150 × 150) due to different surface markings and clutter. Codebook-based object representation and boosted MCMC method can detect large objects robustly. Experimental results validate convincing detection for large objects.
Image-Optimized Rolling Cache: Reducing the Miss Penalty for Memory-Intensive Vision Algorithms

In real-time memory-intensive image processing and vision applications, increasing image resolution requires the use of external SDR/DDR memories. However, the arbitrary pixel access patterns used in most algorithms reduce their memory throughput as a result of increasing access latency. Efficient cache design is paramount in real-time memory-intensive applications. Its effectiveness depends on the spatial and temporal locality of data access. In image processing, the spatial locality denotes the neighboring pixels, located horizontally and vertically in 2-D. However, the conventional caches used in general processors cannot define the vertical locality. We propose a rolling cache optimized for image format and algorithms, a method to reduce the miss penalty by moving the cache horizontally and vertically, and a parallel processing architecture with interpolation, multilevel and multiple caches. To support our idea, we compare it with other types of caches and show that the average memory access time and the memory bandwidth are decreased by 28% and 74%, respectively, for a 2048 × 2048 image. Its performance is greater than that of the 16-way set associative cache, but the tag memory size is a bit larger than that of the direct-mapped cache. Using two different applications, we show that the proposed architecture is applicable to a number of algorithms if data access follows an arbitrary curve or block-wise pattern, which is the usual case with image processing and vision algorithms. If an application is based on local data access in resource-limited systems, it is possible to achieve high performance with lower operational frequency using the proposed architecture.
On-Line Initialization and Extrinsic Calibration of an Inertial Navigation System With a Relative Preintegration Method on Manifold

Inertial measurement units (IMUs) are successfully utilized to compensate localization errors in sensor fused inertial navigation systems. An IMU generally produces high-frequency signals ranging from 100 to 1000 Hz, and preintegration methods are applied to effectively process these high-frequency signals for inertial navigation systems. The main problem with an existing preintegration method is that the inertial propagation models in the method are only generated at the IMU’s coordinate system. Hence, the models have to be converted to the coordinate system of the other sensor in order to apply its constraint. So, the iterative optimization framework using the conventional method takes large amount of time. In addition, since a general rigid body transformation cannot transfer a velocity propagation model to the other coordinate system, the concept of relative motion analysis needs to be considered. To solve the problems above, in this paper, we propose a novel relative preintegration (RP) method that can generate inertial propagation models at any sensor’s coordinate system in a rigid body. This permits accurate and fast IMU processing in sensor fused inertial navigation systems. We applied new nonlinear optimization frameworks to solve initialization and extrinsic calibration problems for the IMU–IMU, IMU–Camera, and IMU–LiDAR pair based on the proposed RP method in an on-line manner, and the superior results of the mentioned processes are presented as well. Note to Practitioners—In the proposed method, extrinsic calibration parameters can be estimated during the localization process without any initial manual precalibration procedure. Besides, an IMU is small and inexpensive, so small AGVs in warehouses, or mobile manipulators in factories are good applications for the proposed method, where they require a certain level of localization accuracy. The localization module using the proposed method can be mounted anywhere in a robot without restrictions. The initialization procedure has been executed for a few seconds with arbitrary movement in the field of view of the first frame, then the initial gravity and velocity vector, and extrinsic parameters, which are estimated during the initialization phase, are applied to a general SLAM problem.
Efficient feature tracking for scene recognition using angular and scale constraints

Recently, many vision-based robotic applications such as visual SLAM (Simultaneous Localization And Mapping) and autonomous navigation have achieved good performance using visual features. In these applications, robust feature tracking plays an important role, e.g., in scene recognition for autonomous navigation and in data association for visual SLAM. In this paper, we propose a hierarchical outlier detection algorithm for robust feature tracking; the algorithm uses a simple window-based correlation (NCC) and enforces angular and scale constraints. The proposed algorithm maximizes the inter-cluster score and detects outliers that do not satisfy the angular constraints. The remaining outliers are detected by enforcing scale constraints using SIFT descriptors. The proposed algorithm is efficient and particularly useful for scene recognition, in which an image corresponding to a query image is searched among data images. Experimental results demonstrate that the proposed algorithm is robust to outliers and image variations such as scale changes. One of the main applications of the proposed algorithm is global localization due to its low computational complexity and robustness to outliers.
Geometry Guided Three-Dimensional Propagation for Depth From Small Motion

In this letter, we present an accurate Depth from Small Motion approach, which reconstructs three-dimensional (3-D) depth from image sequences with extremely narrow baselines. We start with estimating sparse 3-D points and camera poses via the structure from motion method. For dense depth reconstruction, we propose a novel depth propagation using a geometric guidance term that considers not only the geometric constraint from the surface normal, but also color consistency. In addition, we propose an accurate surface normal estimation method with a multiple range search so that the normal vector can guide the direction of the depth propagation precisely. The major benefit of our depth propagation method is that it obtains detailed structures of a scene without fronto-parallel bias. We validate our method using various indoor and outdoor datasets, and both qualitative and quantitative experimental results show that our new algorithm consistently generates better 3-D depth information than the results of existing state-of-the-art methods.
Crosswalk and traffic light detection via integral framework

This research is conducted to detect crosswalks and traffic lights with small false positive and negative errors. We propose an integral framework of the algorithms. Crosswalk and traffic light detection based on vision are challenging problem because brightness and hue value of scene can be easily changed, and it causes a number of false positive and negative errors. In order to solve it, we integrate those two algorithms with an assumption that traffic light and crosswalk exist together frequently. The proposed algorithm is tested with movies captured by cameras located on a vehicle and results are compared with ground truth which was chosen manually. The proposed algorithms run in real-time and can be easily installed on vehicle.
Fast randomized Singular Value Thresholding for Nuclear Norm Minimization

Rank minimization problem can be boiled down to either Nuclear Norm Minimization (NNM) or Weighted NNM (WNNM) problem. The problems related to NNM (or WNNM) can be solved iteratively by applying a closed-form proximal operator, called Singular Value Thresholding (SVT) (or Weighted SVT), but they suffer from high computational cost to compute a Singular Value Decomposition (SVD) at each iteration. In this paper, we propose an accurate and fast approximation method for SVT, called fast randomized SVT (FRSVT), where we avoid direct computation of SVD. The key idea is to extract an approximate basis for the range of a matrix from its compressed matrix. Given the basis, we compute the partial singular values of the original matrix from a small factored matrix. While the basis approximation is the bottleneck, our method is already severalfold faster than thin SVD. By adopting a range propagation technique, we can further avoid one of the bottleneck at each iteration. Our theoretical analysis provides a stepping stone between the approximation bound of SVD and its effect to NNM via SVT. Along with the analysis, our empirical results on both quantitative and qualitative studies show our approximation rarely harms the convergence behavior of the host algorithms. We apply it and validate the efficiency of our method on various vision problems, e.g. subspace clustering, weather artifact removal, simultaneous multi-image alignment and rectification.
A Closed-Form Solution to Rotation Estimation for Structure from Small Motion

The introduction of small motion techniques such as small angle rotation approximation has enabled the three-dimensional reconstruction from a small motion of a camera, so-called structure from small motion (SfSM). In this letter, we propose a closed-form solution dedicated to the rotation estimation problem in SfSM. We show that our method works with a minimal set of two points, and has mild conditions to produce a unique optimal solution in practice. Also, we introduce a three-step SfSM pipeline with better convergence and faster speed compared to the state-of-the-art SfSM approaches. The key to this improvement is the separated estimation of the rotation with the proposed two-point method in order to handle the bas-relief ambiguity that affects the convergence of the bundle adjustment. We demonstrate the effectiveness of our two-point minimal solution and the three-step SfSM approach in synthetic and real-world experiments under the small motion regime.