nils-herrmann/clean_scirepeval huggingface dataset:
Matching shape sequences in video with applications in human movement analysis

We present an approach for comparing two sequences of deforming shapes using both parametric models and nonparametric methods. In our approach, Kendall's definition of shape is used for feature extraction. Since the shape feature rests on a non-Euclidean manifold, we propose parametric models like the autoregressive model and autoregressive moving average model on the tangent space and demonstrate the ability of these models to capture the nature of shape deformations using experiments on gait-based human recognition. The nonparametric model is based on dynamic time-warping. We suggest a modification of the dynamic time-warping algorithm to include the nature of the non-Euclidean space in which the shape deformations take place. We also show the efficacy of this algorithm by its application to gait-based human recognition. We exploit the shape deformations of a person's silhouette as a discriminating feature and provide recognition results using the nonparametric model. Our analysis leads to some interesting observations on the role of shape and kinematics in automated gait-based person authentication.
Diversity-Aware Multi-Video Summarization

Most video summarization approaches have focused on extracting a summary from a single video; we propose an unsupervised framework for summarizing a collection of videos. We observe that each video in the collection may contain some information that other videos do not have, and thus exploring the underlying complementarity could be beneficial in creating a diverse informative summary. We develop a novel diversity-aware sparse optimization method for multi-video summarization by exploring the complementarity within the videos. Our approach extracts a multi-video summary, which is both interesting and representative in describing the whole video collection. To efficiently solve our optimization problem, we develop an alternating minimization algorithm that minimizes the overall objective function with respect to one video at a time while fixing the other videos. Moreover, we introduce a new benchmark data set, Tour20, that contains 140 videos with multiple manually created summaries, which were acquired in a controlled experiment. Finally, by extensive experiments on the new Tour20 data set and several other multi-view data sets, we show that the proposed approach clearly outperforms the state-of-the-art methods on the two problems—topic-oriented video summarization and multi-view video summarization in a camera network.
Learning a geometry integrated image appearance manifold from a small training set

While low-dimensional image representations have been very popular in computer vision, they suffer from two limitations: (i) they require collecting a large and varied training set to learn a low-dimensional set of basis functions, and (ii) they do not retain information about the 3D geometry of the object being imaged. In this paper, we show that it is possible to estimate low-dimensional manifolds that describe object appearance while retaining the geometrical information about the 3D structure of the object. By using a combination of analytically derived geometrical models and statistical learning methods, this can be achieved using a much smaller training set than most of the existing approaches. Specifically, we derive a quadrilinear manifold of object appearance that can represent the effects of illumination, pose, identity and deformation, and the basis functions of the tangent space to this manifold depend on the 3D surface normals of the objects. We show experimental results on constructing this manifold and how to efficiently track on it using an inverse compositional algorithm.
Optimizing hardware design for Human Action Recognition

Human action recognition (HAR) is an important topic in computer vision having a wide range of applications: health care, assisted living, surveillance, security, gaming, etc. Despite significant amount of work having been conducted in this area in recent years, the execution speed still limits real-time applications. Moreover, it is highly desirable to have the compute-intensive feature extraction stage done right at the output of the camera to extract and transfer only action feature in multi-camera network setting and hence reduce network bandwidth requirement. In this work, we first evaluate the possibility to perform feature extraction under reduced precision fixed-point arithmetic to ease hardware resource requirements. We compared the Histogram of Oriented Gradient in 3D (HOG3D) feature extraction with state-of-the-art Convolutional Neural Networks (CNNs) methods and shown the later to be 75× slower than the former. Our experiment shows that by re-training the classifier with reduced data precision, the classification performs as well as the original double-precision floating-point. Based on this result, we implement an FPGA-based HAR feature extraction for near camera processing using fixed-point data representation and arithmetic. This implementation, using a single Xilinx Virtex 6 FPGA, achieves about 70× speedup over multicore CPU. Furthermore, a GPU implementation of HAR is introduced with 80× speedup over CPU (on an Nvidia Tesla K20). Last but not least, a power comparison is presented for the three platforms.
Belief consensus for distributed action recognition

In this work, we consider a camera network where processing is distributed across the cameras. Our goal is to recognize actions of multiple targets consistently observed over the entire network. To obtain consistent and better results we need to properly fuse the action scores from multiple cameras. There have been multiple works on distributed tracking and distributed data association for multiple targets in a camera network. We can use the data association results and tracking confidence scores to improve the action recognition results. We propose a consensus based framework for solving this problem in an integrated manner and with a completely distributed camera network architecture. We propose a novel method for weighting the action scores based on tracking confidences and show how the cameras can reach a consensus about the action of a target using belief consensus. We show real life experiments and performance metrics with multiple cameras and targets.
Consensus-based distributed estimation in camera networks

Distributed algorithms in the sensors networks community usually require each sensor to have its own measurement. In practice, this constraint can not always be met. For example, in a camera network, all cameras might not observe a particular target as cameras are directional sensors and have a limited field-of-view (FOV). Moreover, different sensors might provide different quality measures related to different elements of the measurement vector depending on various factors as directionality, occlusion etc. This requires the designing of a new type of distributed algorithm that considers the quality and/or absence of measurements. In this paper, we present a distributed algorithm to compute the maximum likelihood estimate of the state of a target viewed by the network of cameras, taking into account the above-mentioned factors. We provide step-by-step derivation along with theoretical guarantee of optimality and convergence of the method. Experimental results are provided to show the performance of the proposed algorithm.
Joint Prediction of Activity Labels and Starting Times in Untrimmed Videos

Most of the existing works on human activity analysis focus on recognition or early recognition of the activity labels from complete or partial observations. Predicting the labels of future unobserved activities where no frames of the predicted activities have been observed is a challenging problem, with important applications, which has not been explored much. Associated with the future label prediction problem is the problem of predicting the starting time of the next activity. In this work, we propose a system that is able to infer about the labels and the starting times of future activities. Activities are characterized by the previous activity sequence (which is observed), as well as the objects present in the scene during their occurrence. We propose a network similar to a hybrid Siamese network with three branches to jointly learn both the future label and the starting time. The first branch takes visual features from the objects present in the scene using a fully connected network, the second branch takes previous activity features using a LSTM network to model long-term sequential relationships and the third branch captures the last observed activity features to model the context of inter-activity time using another fully connected network. These concatenated features are used for both label and time prediction. Experiments on two challenging datasets demonstrate that our framework for joint prediction of activity label and starting time improves the performance of both, and outperforms the state-of-the-arts.
Individuals, groups, and crowds: Modelling complex, multi-object behaviour in phase space

This paper concentrates on the problem of modelling and recognition of complex behaviours involving multi-object interactions in video. We use motion patterns of individual objects to construct models which characterize pairs by correlating them in phase space. These models of complex interactions allow for: recognition of group activities, which occur when individual people or objects start interacting as a single entity; detection of transitions from individuals to groups to crowds; and the interactions of individuals with groups, as well as the interactions of groups with other groups. We establish a general formalism by examining activities using relative distances and analyse multi-object interactions directly via the Phase Space Algorithm. Finally, we calculate a scale-invariant Group Transition Ratio to quantify formation and dispersal of both groups and crowds. Our input is solely the position information of individuals, which we get using a person tracker, optical flow, and Lagrangian particle dynamics. We demonstrate the uses of this model for recognition of complex activities on the standard CAVIAR, VIVID, and UCR Videoweb datasets.
Video based rendering of planar dynamic scenes

In this paper, we propose a method to synthesize arbitrary views of a planar scene from a monocular video sequence of it. The 3-D direction of motion of the object is robustly estimated from the video sequence. Given this direction any other view of the object can be synthesized through a perspective projection approach, under assumptions of planarity. If the distance of the object from the camera is large, a planar approximation is reasonable even for non-planar scenes. Such a method has many important applications, one of them being gait recognition where a side view of the person is required. Our method can be used to synthesize the side-view of the person in case he/she does not present a side view to the camera. Since the planarity assumption is often an approximation, the effects of non-planarity can lead to inaccuracies in rendering and needs to be corrected for. Regions where this happens are examined and a simple technique based on weak perspective approximation is proposed to offset rendering inaccuracies. Examples of synthesized views using our method and performance evaluation are presented.
Context-Aware Surveillance Video Summarization

We present a method that is able to find the most informative video portions, leading to a summarization of video sequences. In contrast to the existing works, our method is able to capture the important video portions through information about individual local motion regions, as well as the interactions between these motion regions. In particular, our proposed context-aware video summarization (CAVS) framework adopts the methodology of sparse coding with generalized sparse group lasso to learn a dictionary of video features and a dictionary of spatiotemporal feature correlation graphs. Sparsity ensures that the most informative features and relationships are retained. The feature correlations, represented by a dictionary of graphs, indicate how motion regions correlate with each other globally. When a new video segment is processed by CAVS, both dictionaries are updated in an online fashion. In particular, CAVS scans through every video segment to determine if the new features along with the feature correlations can be sparsely represented by the learned dictionaries. If not, the dictionaries are updated, and the corresponding video segments are incorporated into the summarized video. The results on four public data sets, mostly composed of surveillance videos and a small amount of other online videos, show the effectiveness of our proposed method.
Vector field analysis for motion pattern identification in video

Identification of motion patterns in video is an important problem because it is the first step towards analysis of complex multi-person behaviors to obtain long-term interaction models. In this paper, we will present a flow based technique to identify spatio-temporal motion patterns in a multi-object video. We use the Helmholtz decomposition of optical flow and compute singular points corresponding to component fields. We will show that the optical flow can be used to identify regions which correspond to different moving entities in the video. The singular points in these regions capture the characteristics of the field around them and can be used to identify these regions. This representation would provide us with a framework to analyze activities of individual entities in the scene as well as the global interactions between them. We demonstrate our algorithm on a dataset composed of multi-object videos recorded in a realistic environment.
Information Consensus for Distributed Multi-target Tracking

Due to their high fault-tolerance, ease of installation and scalability to large networks, distributed algorithms have recently gained immense popularity in the sensor networks community, especially in computer vision. Multi-target tracking in a camera network is one of the fundamental problems in this domain. Distributed estimation algorithms work by exchanging information between sensors that are communication neighbors. Since most cameras are directional sensors, it is often the case that neighboring sensors may not be sensing the same target. Such sensors that do not have information about a target are termed as ``naive'' with respect to that target. In this paper, we propose consensus-based distributed multi-target tracking algorithms in a camera network that are designed to address this issue of naivety. The estimation errors in tracking and data association, as well as the effect of naivety, are jointly addressed leading to the development of an information-weighted consensus algorithm, which we term as the Multi-target Information Consensus (MTIC) algorithm. The incorporation of the probabilistic data association mechanism makes the MTIC algorithm very robust to false measurements/clutter. Experimental analysis is provided to support the theoretical results.
An information theoretic criterion for evaluating the quality of 3-D reconstructions from video

Even though numerous algorithms exist for estimating the three-dimensional (3-D) structure of a scene from its video, the solutions obtained are often of unacceptable quality. To overcome some of the deficiencies, many application systems rely on processing more data than necessary, thus raising the question: how is the accuracy of the solution related to the amount of data processed by the algorithm? Can we automatically recognize situations where the quality of the data is so bad that even a large number of additional observations will not yield the desired solution? Previous efforts to answer this question have used statistical measures like second order moments. They are useful if the estimate of the structure is unbiased and the higher order statistical effects are negligible, which is often not the case. This paper introduces an alternative information-theoretic criterion for evaluating the quality of a 3-D reconstruction. The accuracy of the reconstruction is judged by considering the change in mutual information (MI) (termed as the incremental MI) between a scene and its reconstructions. An example of 3-D reconstruction from a video sequence using optical flow equations and known noise distribution is considered and it is shown how the MI can be computed from first principles. We present simulations on both synthetic and real data to demonstrate the effectiveness of the proposed criterion.
A Factorization Approach for Activity Recognition

Understanding activities arising out of the interactions of a configuration of moving objects is an important problem in video understanding, with applications in surveillance and monitoring. A special situation is when the objects are small enough to be represented as points on a 2D plane. In this paper, we introduce a novel method of representing the activity by the deformations of the point configuration in a properly defined shape space. Instead of inferring about the activity directly from the motion tracks of the individual points, we propose to model an activity by the polygonal shape formed by joining the locations of these point masses at any time t, and its deformation as the activity unfolds. Given the locations of the 2D points over a sequence of frames in the video, the factorization theorem for matrices is used to obtain a set of basis shapes for each activity. An unknown activity can now be recognized by projecting onto these basis shapes. Also, once a specific activity is recognized, the deviations from it can be modeled by the deformations from the basis shape. This is used to identify an abnormal activity. We demonstrate the applicability of our algorithm using real-life video sequences in an airport surveillance environment. We are able to identify the major activities that take place in that setting and detect abnormal ones.
Optimized imaging and target tracking within a distributed camera network

This article considers the problem of using a network of NC dynamic pan, tilt, zoom cameras, each mounted at known and fixed locations, to track and obtain high resolution imagery for NT(t) mobile targets each maneuvering within a confined space. The number of targets is time-varying, the targets are free to maneuver, the targets may enter or leave the region under surveillance so that NT(t) is time-varying and may exceed NC. Tracking a target is defined as estimating the position of the target with horizontal uncertainty less that a specified threshold P̅. Imaging a target is defined as obtaining an image with vertical resolution exceeding r̅. The problem is to organize the pan, tilt, and zoom parameters of the network of cameras at each sampling instant such that the tracking specification for all targets and the imaging specification for specific targets at times of opportunity is achieved. This problem could be addressed by centralized or decentralized methods. In this article, we are focused on distributed control of the camera network. We develop a distributed optimization solution, where we consider each camera to be an individual decision making agent. The solution involves formulation of the approach, design of a value function, and design of a probability-based camera ordering mechanism to aid convergence of the distributed network solution towards an optimal solution. Our approach is developed within a Bayesian approach to appropriately trading off value V (target tracking accuracy and target resolution) versus risk (probability of losing track of a target). This article presents the theoretical solution along with simulation results. Implementation on a camera network are in progress.
Stochastic Adaptive Tracking In A Camera Network

We present a novel stochastic, adaptive strategy for tracking multiple people in a large network of video cameras. Similarities between features (appearance and biometrics) observed at different cameras are continuously adapted and the stochastically optimal path for each person computed. The following are the major contributions of the proposed approach. First, we consider situations where the feature similarities are uncertain and treat them as random variables. We show how the distributions of these random variables can be learned and how to compute the tracks in a stochastically optimal manner. Second, we consider the possibility of long-term interdependence of the features over space and time. This allows us to adoptively evolve the feature correspondences by observing the system performance over a time window, and correct for errors in the similarity computations. Third, we show that the above two conditions can be addressed by treating the issue of tracking in a camera network as an optimization problem in a stochastic adaptive system. We show results on data collected by a large camera network. The proposed approach is particularly suitable for distributed processing over the entire network.
Distributed Camera Networks

Over the past decade, large-scale camera networks have become increasingly prevalent in a wide range of applications, such as security and surveillance, disaster response, and environmental modeling. In many applications, bandwidth constraints, security concerns, and difficulty in storing and analyzing large amounts of data centrally at a single location necessitate the development of distributed camera network architectures. Thus, the development of distributed scene-analysis algorithms has received much attention lately. However, the performance of these algorithms often suffers because of the inability to effectively acquire the desired images, especially when the targets are dispersed over a wide field of view (FOV). In this article, we show how to develop an end-to-end framework for integrated sensing and analysis in a distributed camera network so as to maximize various scene-understanding performance criteria (e.g., tracking accuracy, best shot, and image resolution).
Statistical Error Propagation in 3D Modeling From Monocular Video

A significant portion of recent research in computer vision has focused on issues related to sensitivity and robustness of existing techniques. In this paper, we study the classical structure from motion problem and analyze how the statistics representing the quality of the input video propagates through the reconstruction algorithm and affects the quality of the output reconstruction. Specifically, we show that it is possible to derive analytical expressions of the first and second order statistics (bias and error covariance) of the solution as a function of the statistics of the input. We concentrate on the case of reconstruction from a monocular video, where the small baseline makes any algorithm very susceptible to noise in the motion estimates from the video sequence. We derive an expression relating the error covariance of the reconstruction to the error covariance of the feature tracks in the input video. This is done using the implicit function theorem of real analysis and does not require strong statistical assumptions. Next, we prove that the 3D reconstruction is statistically biased, derive an expression for it and show that it is numerically significant. Combining these two results, we also establish a new bound on the minimum error in the depth reconstruction. We present the numerical significance of these analytical results on real video data.