nils-herrmann/clean_scirepeval huggingface dataset:
AVSS 2011 demo session: A large-scale benchmark dataset for event recognition in surveillance video

We present a concept for automatic construction site monitoring by taking into account 4D information (3D over time), that is acquired from highly-overlapping digital aerial images. On the one hand today's maturity of flying micro aerial vehicles (MAVs) enables a low-cost and an efficient image acquisition of high-quality data that maps construction sites entirely from many varying viewpoints. On the other hand, due to low-noise sensors and high redundancy in the image data, recent developments in 3D reconstruction workflows have benefited the automatic computation of accurate and dense 3D scene information. Having both an inexpensive high-quality image acquisition and an efficient 3D analysis workflow enables monitoring, documentation and visualization of observed sites over time with short intervals. Relating acquired 4D site observations, composed of color, texture, geometry over time, largely supports automated methods toward full scene understanding, the acquisition of both the change and the construction site's progress.
Pose and Illumination Invariant Face Recognition in Video

The use of video sequences for face recognition has been relatively less studied than image-based approaches. In this paper, we present a framework for face recognition from video sequences that is robust to large changes in facial pose and lighting conditions. Our method is based on a recently obtained theoretical result that can integrate the effects of motion, lighting and shape in generating an image using a perspective camera. This result can be used to estimate the pose and illumination conditions for each frame of the probe sequence. Then, using a 3D face model, we synthesize images corresponding to the pose and illumination conditions estimated in the probe sequences. Similarity between the synthesized images and the probe video is computed by integrating over the entire sequence. The method can handle situations where the pose and lighting conditions in the training and testing data are completely disjoint.
Super-Resolved Facial Texture Under Changing Pose and Illumination

In this paper, we propose a method to incrementally super-resolve 3D facial texture by integrating information frame by frame from a video captured under changing poses and illuminations. First, we recover illumination, 3D motion and shape parameters from our tracking algorithm. This information is then used to super-resolve 3D texture using iterative back-projection (IBP) method. Finally, the super-resolved texture is fed back to the tracking part to improve the estimation of illumination and motion parameters. This closed-loop process continues to refine the texture as new frames come in. We also propose a local-region based scheme to handle non-rigidity of the human face. Experiments demonstrate that our framework not only incrementally super-resolves facial images, but recovers the detailed expression changes in high quality.
Robust Tracking in A Camera Network: A Multi-Objective Optimization Framework

We address the problem of tracking multiple people in a network of nonoverlapping cameras. This introduces certain challenges that are unique to this particular application scenario, in addition to existing challenges in tracking like pose and illumination variations, occlusion, clutter and sensor noise. For this purpose, we propose a novel multi-objective optimization framework by combining short term feature correspondences across the cameras with long-term feature dependency models. The overall solution strategy involves adapting the similarities between features observed at different cameras based on the long-term models and finding the stochastically optimal path for each person. For modeling the long-term interdependence of the features over space and time, we propose a novel method based on discriminant analysis models. The entire process allows us to adaptively evolve the feature correspondences by observing the system performance over a time window, and correct for errors in the similarity estimations. We show results on data collected by two large camera networks. These experiments prove that incorporation of the long-term models enable us to hold tracks of objects over extended periods of time, including situations where there are large ldquoblindrdquo areas. The proposed approach is implemented by distributing the processing over the entire network.
Context-Aware Activity Modeling Using Hierarchical Conditional Random Fields

In this paper, rather than modeling activities in videos individually, we jointly model and recognize related activities in a scene using both motion and context features. This is motivated from the observations that activities related in space and time rarely occur independently and can serve as the context for each other. We propose a two-layer conditional random field model, that represents the action segments and activities in a hierarchical manner. The model allows the integration of both motion and various context features at different levels and automatically learns the statistics that capture the patterns of the features. With weakly labeled training data, the learning problem is formulated as a max-margin problem and is solved by an iterative algorithm. Rather than generating activity labels for individual activities, our model simultaneously predicts an optimum structural label for the related activities in the scene. We show promising results on the UCLA Office Dataset and VIRAT Ground Dataset that demonstrate the benefit of hierarchical modeling of related activities using both motion and context features.
A Camera Network Tracking (CamNeT) Dataset and Performance Baseline

In this paper, we propose a novel Non-Overlapping Camera Network Tracking Dataset (CamNeT) for evaluating multi-target tracking algorithms. The dataset is composed of five to eight cameras covering both indoor and outdoor scenes at a university. This dataset consists of six scenarios. Within each scenario are challenges relevant to lighting changes, complex topographies, crowded scenes, and changing grouping dynamics. Persons with predefined trajectories are combined with persons with random trajectories. Ground truth data for predefined trajectories is provided for each camera. Also, a baseline multi-target tracking system is presented. The tracking results using the baseline system are provided, which can be compared with future works. The work provides a comprehensive multicamera dataset for performance evaluation in this challenging application domain, as well as an initial set of results.
An Illumination Invariant 3D Model Based Tracking Algorithm, with Application in Video Compression

We present an algorithm for illumination invariant 3D model based tracking and video compression. While model-based coding schemes are well developed, the compression rate reduces if the illumination conditions within the sequence change drastically. Our proposed scheme can accurately track under both slow and drastic changes of lighting and achieves a significant reduction in the bit rate compared to a scheme that does not consider any illumination models. The tracking scheme is based on a recently developed framework showing that the joint motion and illumination space of video sequences is approximately bilinear. We show results of the reconstructed video sequences and the reduction in the distortion (for a constant bit rate) for a compression scheme that uses the illumination invariant tracking algorithm.
A Continuous Learning Framework for Activity Recognition Using Deep Hybrid Feature Models

Most of the research on human activity recognition has focused on learning a static model, considering that all the training instances are labeled and present in advance, while in streaming videos new instances continuously arrive and are not labeled. Moreover, these methods generally use application- specific hand-engineered and static feature models, which are not suitable for continuous learning. Some recent approaches on activity recognition use deep-learning-based hierarchical feature models, but the large size of these networks constrain them from being used in continuous learning scenarios. In this work, we propose a continuous activity learning framework for streaming videos by intricately tying together deep hybrid feature models and active learning. This allows us to automatically select the most suitable features and take the advantage of incoming unlabeled instances to improve the existing model incrementally. Given the segmented activities from streaming videos, we learn features in an unsupervised manner using deep hybrid networks, which have the ability to take the advantage of both the local hand-engineered features and the deep model in an efficient way. Additionally, we use active learning to train the activity classifier using a reduced amount of manually labeled instances. Retraining the models with a huge amount of accumulated examples is computationally expensive and not suitable for continuous learning. Hence, we propose a method to select the best subset of these examples to update the models incrementally. We conduct rigorous experiments on four challenging human activity datasets to demonstrate the effectiveness of our framework.
Coordinated sensing and tracking for mobile camera platforms

This work focuses on developing optimal routing and camera control strategies for multiple mobile PTZ camera platforms cooperatively tracking a stochastic target moving on the ground. The vehicles are modeled as planar Dubins vehicles traveling at a fixed speed and height. The target is tracked using an Extended Kalman Consensus Filter that estimates the target location on the ground plane and the associated error covariance. Using dynamic programming, we compute optimal coordinated control policies which minimize the expected fused geolocation error covariance. The output of this process leads to the trajectories of the vehicles and the PTZ settings of the cameras. We show results on both ground and air vehicles tracking a target whose dynamics is modeled by a stochastic difference equation and discuss the implications of the results. The main contribution of the work is in the development of an optimal strategy for simultaneous vehicle navigation and camera parameter selection to maximize the tracking performance of stochastic targets where the risk of losing the target needs to be considered.
Tracking and Activity Recognition Through Consensus in Distributed Camera Networks

Camera networks are being deployed for various applications like security and surveillance, disaster response and environmental modeling. However, there is little automated processing of the data. Moreover, most methods for multicamera analysis are centralized schemes that require the data to be present at a central server. In many applications, this is prohibitively expensive, both technically and economically. In this paper, we investigate distributed scene analysis algorithms by leveraging upon concepts of consensus that have been studied in the context of multiagent systems, but have had little applications in video analysis. Each camera estimates certain parameters based upon its own sensed data which is then shared locally with the neighboring cameras in an iterative fashion, and a final estimate is arrived at in the network using consensus algorithms. We specifically focus on two basic problems - tracking and activity recognition. For multitarget tracking in a distributed camera network, we show how the Kalman-Consensus algorithm can be adapted to take into account the directional nature of video sensors and the network topology. For the activity recognition problem, we derive a probabilistic consensus scheme that combines the similarity scores of neighboring cameras to come up with a probability for each action at the network level. Thorough experimental results are shown on real data along with a quantitative analysis.
Energy Efficient Object Detection in Camera Sensor Networks

A wireless camera network can provide situation awareness information (e.g., humans in distress) in scenarios such as disaster recovery. If such camera sensors are battery operated, sending raw video feeds back to a central controller can be expensive in terms of energy consumption. Further, if all cameras were to use the optimal processing algorithm for object decision, they may also expend unnecessary energy. Stated otherwise, cameras that capture the same objects may not all have to use the optimal algorithm to achieve a desired accuracy, and this can save processing energy costs. In this paper, our objective is to design and implement a framework that can support coordination among cameras to deliver highly accurate detection of objects in an energy efficient way. The framework, which we call EECS (for energy efficient camera sensors), estimates the detection accuracy and energy costs incurred (both the processing and communication costs are taken into account) with each detection algorithm for each camera, and comes up with a choice of cameras for sending information pertaining to the object of interest. This set of cameras and the video processing algorithms that they must use, are chosen so as to minimize the energy expenditures, given a desired detection accuracy. We implement EECS on a camera network built with smartphones, and demonstrate that it reduces the energy consumption by up to 40% while ensuring a object detection accuracy of over 86%.
Learning Long-Term Invariant Features for Vision-Based Localization

Constructing a feature representation invariant to certain types of geometric and photometric transformations is of significant importance in many computer vision applications. In spite of significant effort, developing invariant feature representations remains a challenging problem. Most of the existing representations often fail to satisfy the longterm repeatability requirements of specific applications like vision-based localization, applications whose domain includes significant, non-uniform illumination and environmental changes. To these ends, we explore the use of natural image pairs (i.e. images captured of the same location but at different times) as an additional source of supervision to generate an improved feature representation for the task of vision-based localization. Specifically, we resort to training deep denoising autoencoder, with CNN feature representation of one image in the pair being treated as a noisy version of the other. The resulting system thereby learns localization features which are both discriminative and invariant to illumination and environmental changes. In experiments tailored towards vision-based localization, features generated using the proposed method produced higher matching rates than state-of-the-art image features.
Wide baseline image registration with application to 3-D face modeling

Establishing correspondence between features in two images of the same scene taken from different viewing angles is a challenging problem in image processing and computer vision. However, its solution is an important step in many applications like wide baseline stereo, three-dimensional (3-D) model alignment, creation of panoramic views, etc. In this paper, we propose a technique for registration of two images of a face obtained from different viewing angles. We show that prior information about the general characteristics of a face obtained from video sequences of different faces can be used to design a robust correspondence algorithm. The method works by matching two-dimensional (2-D) shapes of the different features of the face (e.g., eyes, nose etc.). A doubly stochastic matrix, representing the probability of match between the features, is derived using the Sinkhorn normalization procedure. The final correspondence is obtained by minimizing the probability of error of a match between the entire constellation of features in the two sets, thus taking into account the global spatial configuration of the features. The method is applied for creating holistic 3-D models of a face from partial representations. Although this paper focuses primarily on faces, the algorithm can also be used for other objects with small modifications.
Activity recognition by integrating the physics of motion with a Neuromorphic model of perception

In this paper, we propose a computational framework for integrating the physics of motion with the neurobiological basis of perception in order to model and recognize human actions and object activities. The essence, or gist, of an action is intrinsically related to the motion of the scene's objects. We define the Hamiltonian Energy Signature (HES) and derive the S-Metric to yield a global representation of the motion of the scene's objects in order to capture the gist of the activity. The HES is a scalar time-series that represents the motion of an object over the course of an activity and the S-Metric is a distance metric which characterizes the global motion of the object, or the entire scene, with a single, scalar value. The neurobiological aspect of activity recognition is handled by casting our analysis within a framework inspired by Neuromorphic Computing (NMC), in which we integrate a Motion Energy model with a Form/Shape model. We employ different Form/Shape representations depending on the video resolution but use our HES and S-Metric for the Motion Energy approach in either case. As the core of our Integration mechanism, we utilize variants of the latest neurobiological models of feature integration and biased competition, which we implement within a Multiple Hypothesis Testing (MHT) framework. Experimental validation of the theory is provided on standard datasets capturing a variety of problem settings: single agent actions (KTH), multi-agent actions, and aerial sequences (VIVID).
Face recognition based on SIGMA sets of image features

Automatic face recognition is prevalent in a wide range of systems these days and it is critical to explore new techniques in order to enhance the state of the art. In this paper, we analyze the Region Covariance Matrix (RCM) and its enhancement based on Sigma sets as a feature extraction procedure for face images. The RCM features encode the covariance of various low level features, e.g., pixel intensities and gradients. Sigma sets, on the other hand, reduce the computational complexity of comparing two RCMs. Based on our experiments on the Labeled Faces in the Wild (LFW) dataset, we show that the proposed technique outperforms the popular Local Binary Patterns (LBP) technique and is on par with other better performing techniques that use complex classifiers.
Information weighted consensus

Consensus-based distributed estimation schemes are becoming increasingly popular in sensor networks due to their scalability and fault tolerance capabilities. In a consensus-based state estimation framework, multiple neighboring nodes iteratively communicate with each other, exchanging their own local estimates of a target's state with the goal of converging to a single state estimate over the entire network. However, the state estimation problem becomes challenging when a node has limited observability of the state. In addition, the consensus estimate is sub-optimal when the cross-covariances between the individual state estimates across different nodes are not incorporated in the distributed estimation framework. The cross-covariance is usually neglected because the computational and bandwidth requirements for its computation grow exponentially with the number of nodes. These limitations can be overcome by noting that, as the state estimates at different nodes converge, the information at each node becomes redundant. This fact can be utilized to compute the optimal estimate by proper weighting of the prior state and measurement information. Motivated by this idea, we propose information-weighted consensus algorithms for distributed maximum a posteriori parameter estimates, and their extension to the information-weighted consensus filter (ICF) for state estimation. We show both theoretically and experimentally that the proposed methods asymptotically approach the optimal centralized performance. Simulation results show that ICF is robust even when the optimality conditions are not met and has low communication requirements.
Video synthesis of arbitrary views for approximately planar scenes

In this paper, we propose a method to synthesize arbitrary views of a planar scene, given a monocular video sequence. The method is based on the availability of knowledge of the angle between the original and synthesized views. Such a method has many important applications, one of them being gait recognition. Gait recognition algorithms rely on the availability of an approximate side-view of the person. From a realistic viewpoint, such an assumption is impractical in surveillance applications and it is of interest to develop methods to synthesize a side view of the person, given an arbitrary view. For large distances from the camera, a planar approximation for the individual can be assumed. In this paper, we propose a perspective projection approach for recovering the direction of motion of the person purely from the video data, followed by synthesis of a new video sequence at a different angle. The algorithm works purely in the image and video domain, though 3D structure plays an implicit role in its theoretical justification. Examples of synthesized views using our method and performance evaluation are presented.
Incremental Activity Modeling and Recognition in Streaming Videos

Most of the state-of-the-art approaches to human activity recognition in video need an intensive training stage and assume that all of the training examples are labeled and available beforehand. But these assumptions are unrealistic for many applications where we have to deal with streaming videos. In these videos, as new activities are seen, they can be leveraged upon to improve the current activity recognition models. In this work, we develop an incremental activity learning framework that is able to continuously update the activity models and learn new ones as more videos are seen. Our proposed approach leverages upon state-of-the-art machine learning tools, most notably active learning systems. It does not require tedious manual labeling of every incoming example of each activity class. We perform rigorous experiments on challenging human activity datasets, which demonstrate that the incremental activity modeling framework can achieve performance very close to the cases when all examples are available a priori.
Weakly Supervised Summarization of Web Videos

Most of the prior works summarize videos by either exploring different heuristically designed criteria in an unsupervised way or developing fully supervised algorithms by leveraging human-crafted training data in form of video-summary pairs or importance annotations. However, unsupervised methods are blind to the video category and often fail to produce semantically meaningful video summaries. On the other hand, acquisition of large amount of training data in supervised approaches is non-trivial and may lead to a biased model. Different from existing methods, we introduce a weakly supervised approach that requires only video-level annotation for summarizing web videos. Casting the problem as a weakly supervised learning problem, we propose a flexible deep 3D CNN architecture to learn the notion of importance using only video-level annotation, and without any human-crafted training data. Specifically, our main idea is to leverage multiple videos of a category to automatically learn a parametric model for categorizing videos and then adopt the model to find important segments from a given video as the ones which have maximum influence to the model output. Furthermore, to unleash the full potential of our 3D CNN architecture, we also explored a series of good practices to reduce the influence of limited training data while summarizing videos. Experiments on two challenging and diverse datasets well demonstrate that our approach produces superior quality video summaries compared to several recently proposed approaches.