nils-herrmann/clean_scirepeval huggingface dataset:
Role of shape and kinematics in human movement analysis

Human gait and activity analysis from video is presently attracting a lot of attention in the computer vision community. In this paper we analyze the role of two of the most important cues in human motion-shape and kinematics. We present an experimental framework whereby it is possible to evaluate the relative importance of these two cues in computer vision based recognition algorithms. In the process, we propose a new gait recognition algorithm by computing the distance between two sequences of shapes that lie on a spherical manifold. In our experiments, shape is represented using Kendall's definition of shape. Kinematics is represented using a Linear Dynamical system We place particular emphasis on human gait. Our conclusions show that shape plays a role which is more significant than kinematics in current automated gait based human identification algorithms. As a natural extension we study the role of shape and kinematics in activity recognition. Our experiments indicate that we require models that contain both shape and kinematics in order to perform accurate activity classification. These conclusions also allow us to explain the relative performance of many existing methods in computer-based human activity modeling.
Decentralized camera network control using game theory

This paper deals with the problem of decentralized, cooperative control of a camera network. We focus on applications where events unfold over a large geographic area and need to be analyzed by multiple cameras or other kinds of imaging sensors. There is no central unit accumulating and analyzing all the data. The overall goal is to keep track of all objects (i.e., targets) in the region of deployment of the cameras, while selectively focusing at a high resolution on some particular target features based on application requirements. Efficient usage of resources in such a scenario requires that the cameras be active. However, this control cannot be based on separate analysis of the sensed video in each camera. They must act collaboratively to be able to acquire multiple targets at different resolutions. Our research focuses on developing accurate and efficient target acquisition and camera control algorithms in such scenarios using game theory. We show simulated experimental results of the approach.
Evaluation and Acceleration of High-Throughput Fixed-Point Object Detection on FPGAs

Reliance on object or people detection is rapidly growing beyond surveillance to industrial and social applications. The histogram of oriented gradients (HOG), one of the most popular object detection algorithms, achieves high detection accuracy but delivers just under 1 frame/s on a high-end CPU. Field-programmable gate array (FPGA) accelerations of this algorithm are limited by the intensive floating-point computations. All current fixed-point HOG implementations use large bit width to maintain detection accuracy, or perform poorly at reduced data precision. In this paper, we introduce the full-image evaluation methodology to explore the FPGA implementation of HOG using reduced bit width. This approach lessens the required area resources on the FPGA, and increases the clock frequency and hence the throughput per device through increased parallelism. We evaluate the detection accuracy of the fixed-point HOG by applying state-of-the-art computer vision pedestrian detection evaluation metrics and show it performs as well as the original floating-point code from OpenCV. We then show our single FPGA implementation achieves a 68.7 × higher throughput than a highend CPU, 5.1 × higher than a high-end graphics processing unit (GPU), and 7.8 × higher than the same implementation using floating-point on the same FPGA. A power consumption comparison for different platforms shows our fixed-point FPGA implementation uses 130 × less power than CPU, and 31 × less energy than GPU to process one image.
3D face reconstruction from video using a generic model

Reconstructing a 3D model of a human face from a video sequence is an important problem in computer vision, with applications to recognition, surveillance, multimedia etc. However, the quality of 3D reconstructions using structure from motion (SfM) algorithms is often not satisfactory. One common method of overcoming this problem is to use a generic model of a face. Existing work using this approach initializes the reconstruction algorithm with this generic model. The problem with this approach is that the algorithm can converge to a solution very close to this initial value, resulting in a reconstruction which resembles the generic model rather than the particular face in the video which needs to be modeled. We propose a method of 3D reconstruction of a human face from video in which the 3D reconstruction algorithm and the generic model are handled separately. A 3D estimate is obtained purely from the video sequence using SfM algorithms without use of the generic model. The final 3D model is obtained after combining the SfM estimate and the generic model using an energy function that corrects for the errors in the estimate by comparing local regions in the two models. The optimization is done using a Markov chain Monte Carlo (MCMC) sampling strategy. The main advantage of our algorithm over others is that it is able to retain the specific features of the face in the video sequence even when these features are different from those of the generic model. The evolution of the 3D model through the various stages of the algorithm is presented.
A poisson process model for activity forecasting

Activity forecasting has recently become an active research area for its importance in critical applications like automated navigation and human-computer interaction. However, for a video observed upto a certain time, all of the existing forecasting works focus on predicting the activity label, i.e., predicting what the next unobserved activity is. To the best of our knowledge, no work has answered the crucial question yet: when the next unobserved activity will occur. In this paper, we propose an approach for predicting the starting time of the next unobserved activity without assuming that we know its label. We model activities occurring at a variable rate using a Log-Gaussian Cox Process (LGCP) and learn the rate function from the training data. Then the starting time is predicted using importance sampling algorithm. In our experiments on the challenging MPII-Cooking dataset, we find that both the label of the last observed activity and the label of the activity being predicted affect the time prediction accuracy.
Cell Resolution 3D Reconstruction of Developing Multilayer Tissues from Sparsely Sampled Volumetric Microscopy Images

Understanding of the growth dynamics in developmental biology is often pursued through the analysis of cell sizes and shapes obtained from CLSM based imaging at cell resolution of multi-layer tissues. This necessitates the development of robust 3D reconstruction methods using such images. However, all of the current methods of 3D reconstruction using CLSM imaging require large number of cell slices. But in the case of live cell imaging, i.e., imaging a growing tissue, such high depth resolution is not feasible in order to avoid photodynamic damage to the growing cells from prolonged exposure to laser radiation. In this work, we have addressed the problem of 3D reconstruction at cell resolution of a developing multi-layer tissue in the plant meristem when the amount of data is as limited as two to four slices per cell. This introduces significant image analysis challenges in terms of sparsity of the data, low signal-to-noise ratio, and a wide range of shapes and sizes. Motivated by the physical structure of the cells, we propose to reconstruct a cell cluster as a packing of truncated ellipsoids representing the individual cells. We test the proposed computational method on time-lapse CLSM images of Shoot Apical Meristem (SAM) cells of model plant Arabidopsis Thaliana. We show that the 3D reconstruction can lead to 3D shape models of complete cell clusters, which is an essential first step towards obtaining growth statistics for individual cells.
Online Social Behavior Modeling for Multi-target Tracking

People are often seen together. We use this simple observation to provide crucial additional information and increase the robustness of a video tracker. The goal of this paper is to show how, in situations where offline training data is not available, a social behavior model (SBM) can be inferred online and then integrated within the tracking algorithm. We start with tracklets (short term confident tracks) obtained using an existing tracker. The SBM, a graphical model, captures the spatio-temporal relationships between the tracklets and is learned online from the video. The final probability of association between the tracklets is obtained by a combination of individual target characteristics (e.g., their appearance), as well as the learned relationship model between them. The entire system is causal whereby the results at any given time depend only upon the part of the video already observed. Experimental results on three state-of-the-art datasets show that, without having access to any offline training data or the entire test video a priori (conditions that may be restrictive for many application domains), our proposed method obtains results similar to those that do impose the above conditions.
Hierarchical Graphical Models for Simultaneous Tracking and Recognition in Wide-Area Scenes

We present a unified framework to track multiple people, as well localize, and label their activities, in complex long-duration video sequences. To do this, we focus on two aspects: 1) the influence of tracks on the activities performed by the corresponding actors and 2) the structural relationships across activities. We propose a two-level hierarchical graphical model, which learns the relationship between tracks, relationship between tracks, and their corresponding activity segments, as well as the spatiotemporal relationships across activity segments. Such contextual relationships between tracks and activity segments are exploited at both the levels in the hierarchy for increased robustness. An L1-regularized structure learning approach is proposed for this purpose. While it is well known that availability of the labels and locations of activities can help in determining tracks more accurately and vice-versa, most current approaches have dealt with these problems separately. Inspired by research in the area of biological vision, we propose a bidirectional approach that integrates both bottom-up and top-down processing, i.e., bottom-up recognition of activities using computed tracks and top-down computation of tracks using the obtained recognition. We demonstrate our results on the recent and publicly available UCLA and VIRAT data sets consisting of realistic indoor and outdoor surveillance sequences.
A Neurobiologically Motivated Stochastic Method for Analysis of Human Activities in Video

In this paper, we develop a neurobiologically-motivated statistical method for video analysis that simultaneously searches the combined motion and form space in a concerted and efficient manner using well-known Markov chain Monte Carlo (MCMC) techniques. Specifically, we leverage upon an MCMC variant called the Hamiltonian Monte Carlo (HMC), which we extend to utilize data-based proposals rather than the blind proposals in a traditional HMC, thus creating the Data-Driven HMC (DDHMC). We demonstrate the efficacy of our system on real-life video sequences.
Context-Aware Activity Recognition and Anomaly Detection in Video

In this paper, we propose a mathematical framework to jointly model related activities with both motion and context information for activity recognition and anomaly detection. This is motivated from observations that activities related in space and time rarely occur independently and can serve as context for each other. The spatial and temporal distribution of different activities provides useful cues for the understanding of these activities. We denote the activities occurring with high frequencies in the database as normal activities. Given training data which contains labeled normal activities, our model aims to automatically capture frequent motion and context patterns for each activity class, as well as each pair of classes, from sets of predefined patterns during the learning process. Then, the learned model is used to generate globally optimum labels for activities in the testing videos. We show how to learn the model parameters via an unconstrained convex optimization problem and how to predict the correct labels for a testing instance consisting of multiple activities. The learned model and generated labels are used to detect anomalies whose motion and context patterns deviate from the learned patterns. We show promising results on the VIRAT Ground Dataset that demonstrates the benefit of joint modeling and recognition of activities in a wide-area scene and the effectiveness of the proposed method in anomaly detection.
Multilinear feature extraction and classification of multi-focal images, with applications in nematode taxonomy

In this paper, we present a 3D X-Ray Transform based multilinear feature extraction and classification method for Digital Multi-focal Images (DMI). In such images, morphological information for a transparent specimen can be captured in the form of a stack of high-quality images, representing individual focal planes through the specimen's body. We present a method that can effectively exploit the entire information in the stack using the 3D X-Ray projections at different viewing angles. These DMI stacks represent the effect of different factors - shape, texture, viewpoint, different instances within the same class and different classes of specimens. For this purpose, we embed the 3D X-Ray Transform within a multilinear framework and propose a Multilinear X-Ray Transform (MXRT) feature representation. By combining the tensor texture and shape information we can get better recognition rates than just relying on the original or key frames of DMI stacks. The experimental results on the nematode DMI data show that the 3D X-Ray Transform based multilinear analysis method can effectively give 100% recognition rate on a real-life database.
Multi-target tracking through opportunistic camera control in a resource constrained multimodal sensor network

While wide-area video surveillance is an important application, it is often not practical, from a technical and social perspective, to have video cameras that completely cover the entire region of interest. For obtaining good surveillance results in a sparse camera networks requires that they be complemented by additional sensors with different modalities, their intelligent assignment in a dynamic environment, and scene understanding using these multimodal inputs. In this paper, we propose a probabilistic scheme for opportunistically deploying cameras to the most interesting parts of a scene dynamically given data from a set of video and audio sensors. The audio data is continuously processed to identify interesting events, e.g., entry/exit of people, merging or splitting of groups, and so on. This is used to indicate the time instants to turn on the cameras. Thereafter, analysis of the video determines how long the cameras stay on and whether their pan/tilt/zoom parameters change. Events are tracked continuously by combining the audio and video data. Correspondences between the audio and video sensor observations are obtained through a learned homography between the image plane and ground plane. The method leads to efficient usage of the camera resources by focusing on the most important parts of the scene, saves power, bandwidth and cost, and reduces concerns of privacy. We show detailed experimental results on real data collected in multimodal networks.
Efficient motion estimation under varying illumination

In this paper, we show how to estimate, accurately and efficiently, the 3D motion of a rigid or non-rigid object, and time-varying lighting in a dynamic scene. This is achieved in an inverse compositional tracking framework with a novel warping function that involves a 2D rarr3Drarr2D transformation. The method is guaranteed to converge, is able to work with rigid and non-rigid objects, and estimates the lighting and motion from a video sequence. Experimental analysis on multiple face video sequences shows impressive speed-up over existing methods while retaining a high level of accuracy.
Robust estimation of depth and motion using stochastic approximation

The problem of structure from motion (SfM) is to extract the three-dimensional model of a moving scene from a sequence of images. Though two images are sufficient to produce a 3D reconstruction, they usually perform poorly because of errors in the estimation of the camera motion and image correspondences, thus motivating the need for multiple frame algorithms. One common approach to this problem is to determine the estimate from pairs of images and then fuse them together. Data fusion techniques, like the Kalman filter, require estimates of the error in modeling and observations. The complexity of the SfM problem makes it difficult to reliably estimate these errors. This paper describes a new recursive algorithm to estimate the camera motion and scene structure by fusing the two-frame estimates, using stochastic approximation techniques. The method does not require estimates of the error in the two-frame case and can reconstruct the scene to arbitrary accuracy given a sufficient number of frames. Experimental results are reported to support these claims.
Context-Aware Modeling and Recognition of Activities in Video

In this paper, rather than modeling activities in videos individually, we propose a hierarchical framework that jointly models and recognizes related activities using motion and various context features. This is motivated from the observations that the activities related in space and time rarely occur independently and can serve as the context for each other. Given a video, action segments are automatically detected using motion segmentation based on a nonlinear dynamical model. We aim to merge these segments into activities of interest and generate optimum labels for the activities. Towards this goal, we utilize a structural model in a max-margin framework that jointly models the underlying activities which are related in space and time. The model explicitly learns the duration, motion and context patterns for each activity class, as well as the spatio-temporal relationships for groups of them. The learned model is then used to optimally label the activities in the testing videos using a greedy search method. We show promising results on the VIRAT Ground Dataset demonstrating the benefit of joint modeling and recognizing activities in a wide-area scene.
Graphical models for context-aware analysis of continuous videos

In this paper, we show how graphical models can used for the localization and recognition of activities in continuous videos. The model consists of an action layer and a hidden activity layer. The action layer is modeled as a linear-chain conditional random field (CRF) with the activity labels of action segments as the model variables. Hidden activity variables are then introduced to smooth out the activity labels of action segments and thus generating semantically meaningful activities. With a task-oriented discriminative approach, the learning problem is formulated as a latent Structural Support Vector Machine (SSVM). We show promising results on the UCLA Office Dataset that demonstrate the effectiveness of the proposed framework.
Distributed Constrained Optimization for Bayesian Opportunistic Visual Sensing

We propose a control mechanism to obtain opportunistic high resolution facial imagery, via distributed constrained optimization of the pan, tilt, zoom (PTZ) parameters for each camera in a sensor network. The objective function of the optimization problem quantifies the per camera per target image quality. The tracking constraints, which are a lower bound on the information about the estimated position for each target, define the feasible PTZ parameter space. Each camera alters its own PTZ settings. All cameras use information broadcast by neighboring cameras such that the PTZ parameters of all cameras are simultaneously optimized relative to the global objective. At certain times of opportunity, due to the configuration of the targets relative to the cameras, and the fact that each camera may track many targets, the camera network may be able to reconfigure itself to achieve the tracking specification for all targets with remaining degrees of freedom that can be used to obtain high-res facial images from desirable aspect angles for certain targets. The challenge is to define algorithms to automatically find these time instants, the appropriate imaging camera, and the appropriate parameter settings for all cameras to capitalize on these opportunities. The solution proposed herein involves a Bayesian formulation in a game theoretic setting. The Bayesian formulation automatically trades off objective maximization versus the risk of losing track of any target. This paper describes the problem and solution formulations, design of aligned local and global objective functions and the inequality constraint set, and development of a distributed Lagrangian consensus algorithm that allows cameras to exchange information and asymptotically converge on a pair of primal-dual optimal solutions. This paper presents the theoretical solution along with the simulation results.
Robust estimation of stem cell lineages using local graph matching

In this paper, we present a local graph matching based method for tracking cells and cell divisions. This will allow us to estimate the lineages of the cells in a 4D spatio-temporal image stack obtained using fluorescence imaging techniques. We work with plant cells, where the cells are tightly clustered in space and computing correspondences in space and time can be very challenging. The local graph matching method is able to compute the lineages even when significant portions of the images are corrupted due to sensor noise in the imaging process or segmentation errors. The geometric structure and topology of the cells' relative positions are efficiently exploited to solve the tracking problem using the local graph matching technique. The process not only computes the correspondences of cells across spatial and temporal image slices, but is also able to find out where and when cells divide, identify new cells and detect missing ones. Using this method we show experimental results to track the properly segmented cells, and compute cell lineages from images captured over 72 hours, even when some of those images are highly noisy (e.g., missing cells).
A system identification approach for video-based face recognition

The paper poses video-to-video face recognition as a dynamical system identification and classification problem. We model a moving face as a linear dynamical system whose appearance changes with pose. An autoregressive and moving average (ARMA) model is used to represent such a system. The choice of ARMA model is based on its ability to take care of the change in appearance while modeling the dynamics of pose, expression etc. Recognition is performed using the concept of sub space angles to compute distances between probe and gallery video sequences. The results obtained are very promising given the extent of pose, expression and illumination variation in the video data used for experiments.