nils-herrmann/clean_scirepeval huggingface dataset:
Multi-target tracking using long-term stochastic associations

Maintaining the stability of tracks on multiple targets in video over extended time periods remains a challenging problem. A few methods which have recently shown encouraging results in this direction rely on learning context models or the availability of training data. However, this may not be feasible in many application scenarios. Moreover, tracking methods should be able to work across multiple resolutions of the video. In this paper, we consider the problem of long-term tracking in video in application domains where context information is not available a priori, nor can it be learned online. We build our solution on the hypothesis that most existing trackers can obtain reasonable short-term tracks (tracklets). By analyzing the statistical properties of these tracklets, we develop associations between them so as to come up with longer tracks. On multiple real-life video sequences spanning low and high resolution data, we show the ability to accurately track over extended time periods.
Real Estate Image Classification

Posting pictures is a necessary part of advertising a home for sale. Agents typically sort through dozens of images from which to pick the most complimentary ones. This is a manual effort involving annotating images accompanied by descriptions (bedroom, bathroom, attic, etc.). When volumes are small, manual annotation is not a problem, but there is a point where this becomes too burdensome and ultimately infeasible. Here, we propose an approach based on computer vision methodology to radically increase the efficiency of such tasks. We present a high-confidence image classification framework, whose inputs are images and outputs are labels. The core of the classification algorithm is long short term memory (LSTM), and fully connected neural networks, along with a substantial preprocessing using 'contrast-limited adaptive histogram equalization (CLAHE) for image enhancement. Since, there is no standard benchmark containing a comprehensive dataset of well-annotated real estate images, we introduce Real Estate Image (REI) database for evaluating the image classification algorithms. Therein we demonstrate empirics based on our proposed framework on the new REI dataset, as well as on the SUN dataset.
Collaborative Summarization of Topic-Related Videos

Large collections of videos are grouped into clusters by a topic keyword, such as Eiffel Tower or Surfing, with many important visual concepts repeating across them. Such a topically close set of videos have mutual influence on each other, which could be used to summarize one of them by exploiting information from others in the set. We build on this intuition to develop a novel approach to extract a summary that simultaneously captures both important particularities arising in the given video, as well as, generalities identified from the set of videos. The topic-related videos provide visual context to identify the important parts of the video being summarized. We achieve this by developing a collaborative sparse optimization method which can be efficiently solved by a half-quadratic minimization algorithm. Our work builds upon the idea of collaborative techniques from information retrieval and natural language processing, which typically use the attributes of other similar objects to predict the attribute of a given object. Experiments on two challenging and diverse datasets well demonstrate the efficacy of our approach over state-of-the-art methods.
The Human Action Image

Recognizing a person's motion is intuitive for humans but represents a challenging problem in machine vision. In this paper, we present a multi-disciplinary framework for recognizing human actions. We develop a novel descriptor, the Human Action Image (HAI): a physically-significant, compact representation for the motion of a person, which we derive from first principles in physics using Hamilton's Action. We embed the HAI as the Motion Energy Pathway of the latest Neurobiological model of motion recognition. The Form Pathway is modelled using existing low-level feature descriptors based on shape and appearance. Experimental validation of the theory is provided on the well-known Weizmann and USF Gait datasets.
The Impact of Typicality for Informative Representative Selection

In computer vision, selection of the most informative samples from a huge pool of training data in order to learn a good recognition model is an active research problem. Furthermore, it is also useful to reduce the annotation cost, as it is time consuming to annotate unlabeled samples. In this paper, motivated by the theories in data compression, we propose a novel sample selection strategy which exploits the concept of typicality from the domain of information theory. Typicality is a simple and powerful technique which can be applied to compress the training data to learn a good classification model. In this work, typicality is used to identify a subset of the most informative samples for labeling, which is then used to update the model using active learning. The proposed model can take advantage of the inter-relationships between data samples. Our approach leads to a significant reduction of manual labeling cost while achieving similar or better recognition performance compared to a model trained with entire training set. This is demonstrated through rigorous experimentation on five datasets.
Statistical bias in 3-D reconstruction from a monocular video

The present state-of-the-art in computing the error statistics in three-dimensional (3-D) reconstruction from video concentrates on estimating the error covariance. A different source of error which has not received much attention is the fact that the reconstruction estimates are often significantly statistically biased. In this paper, we derive a precise expression for the bias in the depth estimate, based on the continuous (differentiable) version of structure from motion (SfM). Many SfM algorithms, or certain portions of them, can be posed in a linear least-squares (LS) framework Ax=b. Examples include initialization procedures for bundle adjustment or algorithms that alternately estimate depth and camera motion. It is a well-known fact that the LS estimate is biased if the system matrix A is noisy. In SfM, the matrix A contains point correspondences, which are always difficult to obtain precisely; thus, it is expected that the structure and motion estimates in such a formulation of the problem would be biased. Existing results on the minimum achievable variance of the SfM estimator are extended by deriving a generalized Cramer-Rao lower bound. A detailed analysis of the effect of various camera motion parameters on the bias is presented. We conclude by presenting the effect of bias compensation on reconstructing 3-D face models from rendered images.
Human Identification using Gait and Face

In general the visual-hull approach for performing integrated face and gait recognition requires at least two cameras. In this paper we present experimental results for fusion of face and gait for the single camera case. We considered the NIST database which contains outdoor face and gait data for 30 subjects. In the NIST database, subjects walk along an inverted Sigma pattern. In (A. Kale, et al., 2003), we presented a view-invariant gait recognition algorithm for the single camera case along with some experimental evaluations. In this chapter we present the results of our view-invariant gait recognition algorithm in (A. Kale, et al., 2003) on the NIST database. The algorithm is based on the planar approximation of the person which is valid when the person walks far away from the camera. In (S. Zhou et al., 2003), an algorithm for probabilistic recognition of human faces from video was proposed and the results were demonstrated on the NIST database. Details of these methods can be found in the respective papers. We give an outline of the fusion strategy here.
Inter-dependent CNNs for joint scene and object recognition

In this paper, we consider two inter-dependent deep networks, where one network taps into the other, to perform two challenging cognitive vision tasks - scene classification and object recognition jointly. Recently, convolutional neural networks have shown promising results in each of these tasks. However, as scene and objects are interrelated, the performance of both of these recognition tasks can be further improved by exploiting dependencies between scene and object deep networks. The advantages of considering the inter-dependency between these networks are the following: 1. improvement of accuracy in both scene and object classification, and 2. significant reduction of computational cost in object detection. In order to formulate our framework, we employ two convolutional neural networks (CNNs), scene-CNN and object-CNN. We utilize scene-CNN to generate object proposals which indicate the probable object locations in an image. Object proposals found in the process are semantically relevant to the object. More importantly, the number of object proposals is fewer in amount when compared to other existing methods which reduces the computational cost significantly. Thereafter, in scene classification, we train three hidden layers in order to combine the global (image as a whole) and local features (object information in an image). Features extracted from CNN architecture along with the features processed from object-CNN are combined to perform efficient classification. We perform rigorous experiments on five datasets to demonstrate that our proposed framework outperforms other state-of-the-art methods in classifying scenes as well as recognizing objects.
Active image pair selection for continuous person re-identification

Most traditional multi-camera person re-identification systems rely on learning a static model on tediously labeled training data. Such a framework may not be suitable for situations when new data arrives continuously or all the data is not available for labeling beforehand. Inspired by the `value of information' active learning framework, we propose a continuous learning person re-identification system with a human in the loop. In brief, we term this `continuous person re-identification'. The human in the loop not only provides labels to the incoming images but also improves the learned model by providing most appropriate attribute based explanations. These attribute based explanations are used to learn attribute predictors along the way. The overall effect of such a stratgey is that starting with a few annotated images, the system begins to improve via a symbiotic relationship between the man and the machine. The machine assists the human to speed the annotation and the human assists the machine to update itself with more annotation so that more and more distinct persons are re-identified as more and more images come in. Using a benchmark dataset, we validate our approach and compare with state-of-the-art methods.
A Generalized Kalman Consensus Filter for wide-area video networks

Distributed analysis of video captured by a large network of cameras has received significant attention lately. Tracking moving targets is one of the most fundamental tasks in this regard and the well-known Kalman Consensus Filter (KCF) has been applied to this problem. However, existing solutions do not consider the specific characteristics of video sensor networks, which are necessary for robustness across various application scenarios. Cameras are directional sensors with limited sensing range (field-of-view), and thus, targets are often not observed by many of the cameras. The network may also be spread over a wide area, preventing direct communication between all of the cameras. This limited field-of-view, combined with sparse communication and coverage topologies, motivates us to propose modifications to the traditional KCF framework. Specifically, we consider the covariance matrices of the state estimates of the neighbors and compute a weighted average consensus estimate at each node. Also, the update at each node is computed in two steps, first towards the weighted consensus estimate and then towards the final Kalman measurement update. This leads us to propose a Generalized KCF herein. Experimental results clearly show the advantage of the GKCF compared to the KCF in the considered application scenario.
Integrating Motion, Illumination, and Structure in Video Sequences with Applications in Illumination-Invariant Tracking

In this paper, we present a theory for combining the effects of motion, illumination, 3D structure, albedo, and camera parameters in a sequence of images obtained by a perspective camera. We show that the set of all Lambertian reflectance functions of a moving object, at any position, illuminated by arbitrarily distant light sources, lies "close" to a bilinear subspace consisting of nine illumination variables and six motion variables. This result implies that, given an arbitrary video sequence, it is possible to recover the 3D structure, motion, and illumination conditions simultaneously using the bilinear subspace formulation. The derivation builds upon existing work on linear subspace representations of reflectance by generalizing it to moving objects. Lighting can change slowly or suddenly, locally or globally, and can originate from a combination of point and extended sources. We experimentally compare the results of our theory with ground truth data and also provide results on real data by using video sequences of a 3D face and the entire human body with various combinations of motion and illumination directions. We also show results of our theory in estimating 3D motion and illumination model parameters from a video sequence
Adaptive algorithm selection, with applications in pedestrian detection

Computer vision algorithms are known to be extremely sensitive to the environmental conditions in which the data is captured, e.g., lighting conditions and target density. Tuning of parameters or choosing a completely new algorithm is often needed to achieve a certain performance level. In this paper, we focus on this problem and propose a framework to automatically choose the “best” algorithm-parameter combination (often referred to as the best algorithm for simplicity in this paper) for a certain input data. This necessitates developing a mechanism to switch among different algorithms and parameters as the nature of the input video changes. Specifically, our proposed algorithm calculates a similarity function between a test video segment and a training video segment. Similarity between training and test dataset indicates the same algorithm can be applied to both of them. We design a cost function with this similarity measure and a constraint on the number of switches. In the experiments, we apply our algorithm to the problem of pedestrian detection. We show how to adaptively select among 7 algorithm-parameter combinations and provide promising results on 3 publicly available datasets.
Efficient selection of informative and diverse training samples with applications in scene classification

The huge amount of time required to construct a set of labeled images to train a classifier has led researchers to develop algorithms which can identify the most informative training images, such that labelling those will be sufficient to achieve a considerable classification accuracy. In this paper we focus on choosing a subset of the most informative and diverse images based on which the classification model can be learned efficiently. The size of the subset to be chosen is determined by the available budget for manual labeling. Although the problem of identifying the informative images can be solved by active learning algorithms, it will require a set of labeled images for initial model construction, which is not required in our method as we identify the best samples at one shot. We incorporate the concepts of strong and weak teacher to help the learner to learn the model efficiently with limited budget for manual labeling. We perform rigorous experiments on two challenging scene classification datasets to demonstrate the effectiveness of our algorithm.
Facial similarity across age, disguise, illumination and pose

Illumination, pose variations, disguises, aging effects and expression variations are some of the key factors that affect the performance of face recognition systems. Face recognition systems have always been studied from a recognition perspective. Our emphasis is on deriving a measure of similarity between faces. The similarity measure provides insights into the role each of the above mentioned variations play in affecting the performance of face recognition systems. In the process of computing the similarity measure between faces, we suggest a framework to compensate for pose variations and introduce the notion of 'half-faces' to circumvent the problem of non-uniform illumination. We used the similarity measure to retrieve similar faces from a database containing multiple images of individuals. Moreover, we devised experiments to study the effect age plays in affecting facial similarity. In conclusion, the similarity measure helps in studying the significance facial features play in affecting the performance of face recognition systems.
Integrating the effects of motion, illumination and structure in video sequences

Most work in computer vision has concentrated on studying the individual effect of motion and illumination on a 3D object. In this paper, we present a theory for combining the effects of motion, illumination, 3D structure, albedo, and camera parameters in a sequence of images obtained by a perspective camera. We show that the set of all Lambertian reflectance functions of a moving object, illuminated by arbitrarily distant light sources, lies close to a bilinear subspace consisting of nine illumination variables and six motion variables. This result implies that, given an arbitrary video sequence, it is possible to recover the 3D structure, motion and illumination conditions simultaneously using the bilinear subspace formulation. The derivation is based on the intuitive notion that, given an illumination direction, the images of a moving surface cannot change suddenly over a short time period. We experimentally compare the images obtained using our theory with ground truth data and show that the difference is small and acceptable. We also provide experimental results on real data by synthesizing video sequences of a 3D face with various combinations of motion and illumination directions
Learning Temporal Regularity in Video Sequences

Perceiving meaningful activities in a long video sequence is a challenging problem due to ambiguous definition of 'meaningfulness' as well as clutters in the scene. We approach this problem by learning a generative model for regular motion patterns (termed as regularity) using multiple sources with very limited supervision. Specifically, we propose two methods that are built upon the autoencoders for their ability to work with little to no supervision. We first leverage the conventional handcrafted spatio-temporal local features and learn a fully connected autoencoder on them. Second, we build a fully convolutional feed-forward autoencoder to learn both the local features and the classifiers as an end-to-end learning framework. Our model can capture the regularities from multiple datasets. We evaluate our methods in both qualitative and quantitative ways - showing the learned regularity of videos in various aspects and demonstrating competitive performance on anomaly detection datasets as an application.
Non-uniform Subset Selection for Active Learning in Structured Data

Several works have shown that relationships between data points (i.e., context) in structured data can be exploited to obtain better recognition performance. In this paper, we explore a different, but related, problem: how can these inter-relationships be used to efficiently learn and continuously update a recognition model, with minimal human labeling effort. Towards this goal, we propose an active learning framework to select an optimal subset of data points for manual labeling by exploiting the relationships between them. We construct a graph from the unlabeled data to represent the underlying structure, such that each node represents a data point, and edges represent the inter-relationships between them. Thereafter, considering the flow of beliefs in this graph, we choose those samples for labeling which minimize the joint entropy of the nodes of the graph. This results in significant reduction in manual labeling effort without compromising recognition performance. Our method chooses non-uniform number of samples from each batch of streaming data depending on its information content. Also, the submodular property of our objective function makes it computationally efficient to optimize. The proposed framework is demonstrated in various applications, including document analysis, scene-object recognition, and activity recognition.
A robust algorithm for fusing noisy depth estimates using stochastic approximation

The problem of structure from motion (SFM) is to extract the three-dimensional model of a moving scene from a sequence of images. Most of the algorithms which work by fusing the two-frame depth estimates (observations) assume an underlying statistical model for the observations and do not evaluate the quality of the individual observations. However, in real scenarios, it is often difficult to justify the statistical assumptions. Also, outliers are present in any observation sequence and need to be identified and removed from the fusion algorithm. We present a recursive fusion algorithm using the Robbins-Monro stochastic approximation (RMSA) which takes care of both these problems to provide an estimate of the real depth of the scene point. The estimate converges to the true value asymptotically. We also propose a method to evaluate the importance of the successive observations by computing the Fisher information (FI) recursively. Though we apply our algorithm in the SFM problem by modeling of a human face, it can be easily adopted to other data fusion applications.
Determining Topology in a Distributed Camera Network

Recently, 'entry/exit' events of objects in the field-of-views of cameras were used to learn the topology of the camera network. The integration of object appearance was also proposed to employ the visual information provided by the imaging sensors. A problem with these methods is the lack of robustness to appearance changes. This paper integrates face recognition in the statistical model to better estimate the correspondence in the time-varying network. The statistical dependence between the entry and exit nodes indicates the connectivity and traffic patterns of the camera network, which are represented by a weighted directed graph and transition time distributions. A nine-camera network with 25 nodes is analyzed both in simulation and in real-life experiments, and compared with the previous approaches.
A study on view-insensitive gait recognition

Most gait recognition approaches only study human walking frontoparallel to the image plane which is not realistic in video surveillance applications. Human gait appearance depends on various factors including locations of the camera and the person, the camera axis and the walking direction. By analyzing these factors, we propose a statistical approach for view-insensitive gait recognition. The proposed approach recognizes human using a single camera, and avoids the difficulties of recovering the human body structure and camera calibration. Experimental results show that the proposed approach achieves good performance in recognizing individuals walking along different directions.