nils-herrmann/clean_scirepeval huggingface dataset:
Matching Features Correctly through Semantic Understanding

Image-to-image feature matching is the single most restrictive time bottleneck in any matching pipeline. We propose two methods for improving the speed and quality by employing semantic scene segmentation. First, we introduce a way of capturing semantic scene context of a key point into a compact description. Second, we propose to learn correct match ability of descriptors from these semantic contexts. Finally, we further reduce the complexity of matching to only a pre-computed set of semantically close key points. All methods can be used independently and in the evaluation we show combinations for maximum speed benefits. Overall, our proposed methods outperform all baselines and provide significant improvements in accuracy and an order of magnitude faster key point matching.
Optimal Transformation Estimation with Semantic Cues

This paper addresses the problem of estimating the geometric transformation relating two distinct visual modalities (e.g. an image and a map, or a projective structure and a Euclidean 3D model) while relying only on semantic cues, such as semantically segmented regions or object bounding boxes. The proposed approach differs from the traditional feature-to-feature correspondence reasoning: starting from semantic regions on one side, we seek their possible corresponding regions on the other, thus constraining the sought geometric transformation. This entails a simultaneous search for the transformation and for the region-to-region correspondences. This paper is the first to derive the conditions that must be satisfied for a convex region, defined by control points, to be transformed inside an ellipsoid. These conditions are formulated as Linear Matrix Inequalities and used within a Branch-and-Prune search to obtain the globally optimal transformation. We tested our approach, under mild initial bound conditions, on two challenging registration problems for aligning: (i) a semantically segmented image and a map via a 2D homography; (ii) a projective 3D structure and its Euclidean counterpart.
Combined Depth and Outlier Estimation in Multi-View Stereo

In this paper, we present a generative model based approach to solve the multi-view stereo problem. The input images are considered to be generated by either one of two processes: (i) an inlier process, which generates the pixels which are visible from the reference camera and which obey the constant brightness assumption, and (ii) an outlier process which generates all other pixels. Depth and visibility are jointly modelled as a hiddenMarkov Random Field, and the spatial correlations of both are explicitly accounted for. Inference is made tractable by an EM-algorithm, which alternates between estimation of visibility and depth, and optimisation of model parameters. We describe and compare two implementations of the E-step of the algorithm, which correspond to the Mean Field and Bethe approximations of the free energy. The approach is validated by experiments on challenging real-world scenes, of which two are contaminated by independently moving objects.
Analysis of Human Locomotion based on Partial Measurements

A lot of computer vision applications have to deal with occlusions. In such settings only a subset of the features of interest can be observed, i.e. only incomplete or partial measurements are available. In this article we show how a learned statistical model can be used to make a prediction of the unknown (occluded) features. The probabilistic nature of the framework also allows to compute the remaining uncertainty given an incomplete observation. The resulting posterior probability distribution can then be used for inference. Additional unknowns such as alignment or scale are easily incorporated into the framework. Instead of computing the alignment in a preprocessing step, it is left as an additional uncertainty, similar to the uncertainty introduced by the missing values of the measurement. It is shown how the technique can be applied to the analysis of human loco-motion, when body parts are occluded. Experiments show how the unobserved body locations are predicted and how it can be inferred whether the measurements come from a running or walking sequence.
Prime Object Proposals with Randomized Prim's Algorithm

Generic object detection is the challenging task of proposing windows that localize all the objects in an image, regardless of their classes. Such detectors have recently been shown to benefit many applications such as speeding-up class-specific object detection, weakly supervised learning of object detectors and object discovery. In this paper, we introduce a novel and very efficient method for generic object detection based on a randomized version of Prim's algorithm. Using the connectivity graph of an image's super pixels, with weights modelling the probability that neighbouring super pixels belong to the same object, the algorithm generates random partial spanning trees with large expected sum of edge weights. Object localizations are proposed as bounding-boxes of those partial trees. Our method has several benefits compared to the state-of-the-art. Thanks to the efficiency of Prim's algorithm, it samples proposals very quickly: 1000 proposals are obtained in about 0.7s. With proposals bound to super pixel boundaries yet diversified by randomization, it yields very high detection rates and windows that tightly fit objects. In extensive experiments on the challenging PASCAL VOC 2007 and 2012 and SUN2012 benchmark datasets, we show that our method improves over state-of-the-art competitors for a wide range of evaluation scenarios.
Reading between the lines-a method for extracting dynamic 3D with texture

A method is presented that extracts the 3D shape of objects, together with the surface texture. Both shape and texture are obtained from a single image. The underlying principle is based on an active technique. A high resolution, pattern is projected onto the object and the deformations as observed by a single camera yield the 3rd dimension. Furthermore, the surface texture is extracted from the same image. Because the whole procedure is based on a single image, a frame-by-frame reconstruction of a video taken with the pattern projected throughout, yields 3D shape dynamics. The paper sketches the complete system but focuses on the problem of texture extraction.
PathTrack: Fast Trajectory Annotation with Path Supervision

Progress in Multiple Object Tracking (MOT) has been historically limited by the size of the available datasets. We present an efficient framework to annotate trajectories and use it to produce a MOT dataset of unprecedented size. In our novel path supervision the annotator loosely follows the object with the cursor while watching the video, providing a path annotation for each object in the sequence. Our approach is able to turn such weak annotations into dense box trajectories. Our experiments on existing datasets prove that our framework produces more accurate annotations than the state of the art, in a fraction of the time. We further validate our approach by crowdsourcing the PathTrack dataset, with more than 15,000 person trajectories in 720 sequences. Tracking approaches can benefit training on such large-scale datasets, as did object recognition. We prove this by re-training an off-the-shelf person matching network, originally trained on the MOT15 dataset, almost halving the misclassification rate. Additionally, training on our data consistently improves tracking results, both on our dataset and on MOT15. On the latter, we improve the top-performing tracker (NOMT) dropping the number of ID Switches by 18% and fragments by 5%.
UntrimmedNets for Weakly Supervised Action Recognition and Detection

Current action recognition methods heavily rely on trimmed videos for model training. However, it is expensive and time-consuming to acquire a large-scale trimmed video dataset. This paper presents a new weakly supervised architecture, called UntrimmedNet, which is able to directly learn action recognition models from untrimmed videos without the requirement of temporal annotations of action instances. Our UntrimmedNet couples two important components, the classification module and the selection module, to learn the action models and reason about the temporal duration of action instances, respectively. These two components are implemented with feed-forward networks, and UntrimmedNet is therefore an end-to-end trainable architecture. We exploit the learned models for action recognition (WSR) and detection (WSD) on the untrimmed video datasets of THUMOS14 and ActivityNet. Although our UntrimmedNet only employs weak supervision, our method achieves performance superior or comparable to that of those strongly supervised approaches on these two datasets.
Handling Occlusions with Franken-Classifiers

Detecting partially occluded pedestrians is challenging. A common practice to maximize detection quality is to train a set of occlusion-specific classifiers, each for a certain amount and type of occlusion. Since training classifiers is expensive, only a handful are typically trained. We show that by using many occlusion-specific classifiers, we outperform previous approaches on three pedestrian datasets, INRIA, ETH, and Caltech USA. We present a new approach to train such classifiers. By reusing computations among different training stages, 16 occlusion-specific classifiers can be trained at only one tenth the cost of one full training. We show that also test time cost grows sub-linearly.
3D City Modeling Using Cognitive Loops

3D city modeling using computer vision is very challenging. A typical city contains objects which are a nightmare for some vision algorithms, while other algorithms have been designed to identify exactly these parts but, in their turn, suffer from other weaknesses which limit their application. For instance, moving cars with metallic surfaces can degrade the results of a 3D city reconstruction algorithm which is primarily based on the assumption of a static scene with diffuse reflection properties. On the other hand, a specialized object recognition algorithm could be able to detect cars, but also yields too many false positives without the availability of additional scene knowledge. In this paper, the design of a cognitive loop which intertwines both aforementioned algorithms is demonstrated for 3D city modeling, proving that the whole can be much more than the simple sum of its parts. A cognitive loop is the mutual transfer of higher knowledge between algorithms, which enables the combination of algorithms to overcome the weaknesses of any single algorithm. We demonstrate the promise of this approach on a real-world city modeling task using video data recorded by a survey vehicle. Our results show that the cognitive combination of algorithms delivers convincing city models which improve upon the degree of realism that is possible from a purely reconstruction-based approach.
Gesture Recognition Portfolios for Personalization

Human gestures, similar to speech and handwriting, are often unique to the individual. Training a generic classifier applicable to everyone can be very difficult and as such, it has become a standard to use personalized classifiers in speech and handwriting recognition. In this paper, we address the problem of personalization in the context of gesture recognition, and propose a novel and extremely efficient way of doing personalization. Unlike conventional personalization methods which learn a single classifier that later gets adapted, our approach learns a set (portfolio) of classifiers during training, one of which is selected for each test subject based on the personalization data. We formulate classifier personalization as a selection problem and propose several algorithms to compute the set of candidate classifiers. Our experiments show that such an approach is much more efficient than adapting the classifier parameters but can still achieve comparable or better results.
You'll never walk alone: Modeling social behavior for multi-target tracking

Object tracking typically relies on a dynamic model to predict the object's location from its past trajectory. In crowded scenarios a strong dynamic model is particularly important, because more accurate predictions allow for smaller search regions, which greatly simplifies data association. Traditional dynamic models predict the location for each target solely based on its own history, without taking into account the remaining scene objects. Collisions are resolved only when they happen. Such an approach ignores important aspects of human behavior: people are driven by their future destination, take into account their environment, anticipate collisions, and adjust their trajectories at an early stage in order to avoid them. In this work, we introduce a model of dynamic social behavior, inspired by models developed for crowd simulation. The model is trained with videos recorded from birds-eye view at busy locations, and applied as a motion model for multi-people tracking from a vehicle-mounted camera. Experiments on real sequences show that accounting for social interactions and scene knowledge improves tracking performance, especially during occlusions.
Online Video SEEDS for Temporal Window Objectness

Super pixel and objectness algorithms are broadly used as a pre-processing step to generate support regions and to speed-up further computations. Recently, many algorithms have been extended to video in order to exploit the temporal consistency between frames. However, most methods are computationally too expensive for real-time applications. We introduce an online, real-time video super pixel algorithm based on the recently proposed SEEDS super pixels. A new capability is incorporated which delivers multiple diverse samples (hypotheses) of super pixels in the same image or video sequence. The multiple samples are shown to provide a strong cue to efficiently measure the objectness of image windows, and we introduce the novel concept of objectness in temporal windows. Experiments show that the video super pixels achieve comparable performance to state-of-the-art offline methods while running at 30 fps on a single 2.8 GHz i7 CPU. State-of-the-art performance on objectness is also demonstrated, yet orders of magnitude faster and extended to temporal windows in video.
Incremental Learning of Random Forests for Large-Scale Image Classification

Large image datasets such as ImageNet or open-ended photo websites like Flickr are revealing new challenges to image classification that were not apparent in smaller, fixed sets. In particular, the efficient handling of dynamically growing datasets, where not only the amount of training data but also the number of classes increases over time, is a relatively unexplored problem. In this challenging setting, we study how two variants of Random Forests (RF) perform under four strategies to incorporate new classes while avoiding to retrain the RFs from scratch. The various strategies account for different trade-offs between classification accuracy and computational efficiency. In our extensive experiments, we show that both RF variants, one based on Nearest Class Mean classifiers and the other on SVMs, outperform conventional RFs and are well suited for incrementally learning new classes. In particular, we show that RFs initially trained with just 10 classes can be extended to 1,000 classes with an acceptable loss of accuracy compared to training from the full data and with great computational savings compared to retraining for each new batch of classes.
Unsupervised workflow discovery in industrial environments

In this work, we present an approach for the automatic discovery of workflows in industrial environments. In such cluttered scenes, one faces many challenges, which limit the use of state-of-the-art object detection and tracking methods. Instead we propose a purely data-driven method which exploits the temporal structure of the workflow. Our robust technique is free of human intervention and does not need parameter tuning. We show results on two camera views of a working cell in a car assembly line. Workflows are extracted robustly, they match well across the camera views and they are conform with human annotation. Furthermore, we show a simple but efficient extension to analyze the image stream in real time. This assures a smooth running of the workflow and enables the notification of different types of unexpected scenarios.
Computational Symmetry in Computer Vision and Computer Graphics

In the arts and sciences, as well as in our daily lives, symmetry has made a profound and lasting impact. Likewise, a computational treatment of symmetry and group theory (the ultimate mathematical formalization of symmetry) has the potential to play an important role in computational sciences. Though the term computational symmetry was formally defined a decade ago by the first author, referring to algorithmic treatment of symmetries, seeking symmetry from digital data has been attempted for over four decades. Computational symmetry on real world data turns out to be challenging enough that, after decades of effort, a fully automated symmetry–savvy system remains elusive for real world applications. The recent resurging interests in computational symmetry for computer vision and computer graphics applications have shown promising results. Recognizing the fundamental relevance and Full text available at: http://dx.doi.org/10.1561/0600000008
Stochastic optimisation for high-dimensional tracking in dense range maps

The main challenge of tracking articulated structures like hands is their many degrees of freedom (DOFs). A realistic 3-D model of the human hand has at least 26 DOFs. The arsenal of tracking approaches that can track such structures fast and reliably is still very small. This paper proposes a tracker based on stochastic meta-descent (SMD) for optimisations in such high-dimensional state spaces. This new algorithm is based on a gradient descent approach with adaptive and parameter-specific step sizes. The SMD tracker facilitates the integration of constraints, and combined with a stochastic sampling technique, can get out of spurious local minima. Furthermore, the integration of a deformable hand model based on linear blend skinning and anthropometrical measurements reinforces the robustness of the tracker. Experiments show the efficiency of the SMD algorithm in comparison with common optimisation methods.
Efficient, Simultaneous Detection of Multiple Object Classes

At present, the object categorisation literature is still dominated by the use of individual class detectors. Detecting multiple classes then implies the subsequent application of multiple such detectors, but such an approach is not scalable towards high numbers of classes. This paper presents an alternative strategy, where multiple classes are detected in a combined way. This includes a decision tree approach, where ternary rather than binary nodes are used, and where nodes share features. This yields an efficient scheme, which scales much better. The paper proposes a strategy where the object samples are first distinguished from the background. Then, in a second stage, the actual object class membership of each sample is determined. The focus of the paper lies entirely on the first stage, i.e. the distinction from background. The tree approach for this step is compared against two alternative strategies, one of them being the popular cascade approach. While classification accuracy tends to be better or comparable, the speed of the proposed method is systematically better. This advantage gets more outspoken as the number of object classes increases
Robust Estimation in the Presence of Spatially Coherent Outliers

We present a generative model based approach to deal with spatially coherent outliers. The model assumes that image pixels are generated by either one of two distinct processes: an inlier process which is responsible for the generation of the majority of the data, and an outlier process which generates pixels not adhering to the inlier model. The partitioning into inlier and outlier regions is made explicit by the introduction of a hidden binary map. To account for the coherent nature of outliers this map is modelled as a Markov Random Field, and inference is made tractable by a mean field EM-algorithm. We make a connection with classical robust estimation theory, and derive the analytic expressions of the equivalent M-estimator for two limiting cases of our model. The effectiveness of the proposed method is demonstrated with two examples. First, in a synthetic linear regression problem, we compare our approach with different M-estimators. Next, in a 2D-face recognition experiment, we try to identify people from partially occluded facial images.
AWEAR 2.0 system: Omni-directional audio-visual data acquisition and processing

We present a wearable audio-visual capturing system, termed AWEAR 2.0, along with its underlying vision components that allow robust self-localization, multi-body pedestrian tracking, and dense scene reconstruction. Designed as a backpack, the system is aimed at supporting the cognitive abilities of the wearer. In this paper, we focus on the design issues for the hardware platform and on the performance of the current state-of-the-art computer vision methods on the acquired sequences. We describe the calibration procedure of the two omni-directional cameras present in the system as well as a structure-from-motion pipeline that allows for stable multi-body tracking even from rather shaky video sequences thanks to ground plane stabilization. Furthermore, we show how a dense scene reconstruction can be obtained from the data acquired with the platform.
Integrating Object Detection with 3D Tracking Towards a Better Driver Assistance System

Driver assistance helps save lives. Accurate 3D pose is required to establish if a traffic sign is relevant to the driver. We propose a real-time system that integrates single view detection with region-based 3D tracking of road signs. The optimal set of candidate detections is found, followed by AdaBoost cascades and SVMs. The 2D detections are then employed in simultaneous 2D segmentation and 3D pose tracking, using the known 3D model of the recognised traffic sign. We demonstrate the abilities of our system by tracking multiple road signs in real world scenarios.
Eliciting qualitative structure from image curve deformations

A family of qualitative methods is described that determines structure of curves in a scene from their image projections in two or more views. For example, plane curves are distinguished from space curves, and space curves from contour generators. The novelty of the approach is that first, it is unaffected by camera intrinsic parameters, so calibration is not required; second, it is also unaffected by camera motion, so viewer motion need not be known; and third, curves need not be matched point by point. Qualitative scene descriptions can be determined using relative measurements, and affine or projective properties alone. The practical power of this family of methods is demonstrated using a real-time vision system based on B-spline snake-tracking.<<ETX>>