nils-herrmann/clean_scirepeval huggingface dataset:
Recognition of Humans and Their Activities Using Video

The recognition of humans and their activities from video sequences is currently a very active area of research because of its applications in video surveillance, design of realistic entertainment systems, multimedia communications, and medical diagnosis. In this lecture, we discuss the use of face and gait signatures for human identification and recognition of human activities from video sequences. We survey existing work and describe some of the more well-known methods in these areas. We also describe our own research and outline future possibilities. In the area of face recognition, we start with the traditional methods for image-based analysis and then describe some of the more recent developments related to the use of video sequences, 3D models, and techniques for representing variations of illumination. We note that the main challenge facing researchers in this area is the development of recognition strategies that are robust to changes due to pose, illumination, disguise, and aging. Gait r...
The Function Space of an Activity

An activity consists of an actor performing a series of actions in a pre-defined temporal order. An action is an individual atomic unit of an activity. Different instances of the same activity may consist of varying relative speeds at which the various actions are executed, in addition to other intra- and inter- person variabilities. Most existing algorithms for activity recognition are not very robust to intra- and inter-personal changes of the same activity, and are extremely sensitive to warping of the temporal axis due to variations in speed profile. In this paper, we provide a systematic approach to learn the nature of such time warps while simultaneously allowing for the variations in descriptors for actions. For each activity we learn an ‘average’ sequence that we denote as the nominal activity trajectory. We also learn a function space of time warpings for each activity separately. The model can be used to learn individualspecific warping patterns so that it may also be used for activity based person identification. The proposed model leads us to algorithms for learning a model for each activity, clustering activity sequences and activity recognition that are robust to temporal, intra- and inter-person variations. We provide experimental results using two datasets.
Computerized Face Recognition in Renaissance Portrait Art: A quantitative measure for identifying uncertain subjects in ancient portraits

This article explores the feasibility of face-recognition technologies for analyzing works of portraiture and, in the process, provides a quantitative source of evidence to art historians in answering many of their ambiguities concerning identity of the subject in some portraits and in understanding artists? styles. Works of portrait art bear the mark of visual interpretation of the artist. Moreover, the number of samples available to model these effects is often limited. Based on an understanding of artistic conventions, we show how to learn and validate features that are robust in distinguishing subjects in portraits (sitters) and that are also capable of characterizing an individual artist?s style. This can be used to learn a feature space called portrait feature space (PFS) that is representative of quantitative measures of similarities between portrait pairs known to represent same/different sitters. Through statistical hypothesis tests, we analyze uncertain portraits against known identities and explain the significance of the results from an art historian?s perspective. Results are shown on our data consisting of over 270 portraits belonging largely to the Renaissance era.
Multi-focal nematode image classification using the 3D X-Ray Transform

In this paper, we present a 3D X-Ray Transform based feature extraction and classification method for Digital Multi-focal Images (DMI). In such images, morphological information for a transparent specimen can be captured in the form of a stack of high-quality images, representing individual focal planes through the specimen's body. We present a method that can effectively exploit the entire information in the stack using the 3D X-Ray Transform at different angle views. By combining the texture and shape information from different angles, we can get better recognition rates than just relying on the original or key frames of DMI stacks. The experimental results on the nematode DMI data show that the 3D X-Ray Transform based classification method can effectively improve the recognition rate from 60% (PCA) to 96.8%.
Summarization and Indexing of Human Activity Sequences

In order to summarize a video consisting of a sequence of different activities, there are three fundamental problems: tracking the objects of interest, detecting the activity change times and recognizing the new activity. This paper presents an algorithm for achieving all these three tasks simultaneously and presents results on how it can used for indexing and summarizing a real-life video sequence. Human activities are represented by a model for the dynamics of the shape of the human body contour. Measures are designed for detecting both gradual transitions and sudden changes between activity models.
Unsupervised Adaptive Re-identification in Open World Dynamic Camera Networks

Person re-identification is an open and challenging problem in computer vision. Existing approaches have concentrated on either designing the best feature representation or learning optimal matching metrics in a static setting where the number of cameras are fixed in a network. Most approaches have neglected the dynamic and open world nature of the re-identification problem, where a new camera may be temporarily inserted into an existing system to get additional information. To address such a novel and very practical problem, we propose an unsupervised adaptation scheme for re-identification models in a dynamic camera network. First, we formulate a domain perceptive re-identification method based on geodesic flow kernel that can effectively find the best source camera (already installed) to adapt with a newly introduced target camera, without requiring a very expensive training phase. Second, we introduce a transitive inference algorithm for re-identification that can exploit the information from best source camera to improve the accuracy across other camera pairs in a network of multiple cameras. Extensive experiments on four benchmark datasets demonstrate that the proposed approach significantly outperforms the state-of-the-art unsupervised learning based alternatives whilst being extremely efficient to compute.
Sparse modeling for topic-oriented video summarization

While most existing video summarization approaches aim to extract an informative summary of a single video, we propose an unsupervised framework for summarizing topic-related videos by exploring complementarity within videos. We develop a novel sparse optimization method to extract a diverse summary that is both interesting and representative in describing the video collection. To efficiently solve our optimization problem, we develop an alternating minimization algorithm that minimizes the overall objective function with respect to one video at a time while fixing the other videos. Experimental results demonstrate that our approach clearly outperforms the state-of-the-art methods.
"Shape Activity": a continuous-state HMM for moving/deforming shapes with application to abnormal activity detection

The aim is to model "activity" performed by a group of moving and interacting objects (which can be people, cars, or different rigid components of the human body) and use the models for abnormal activity detection. Previous approaches to modeling group activity include co-occurrence statistics (individual and joint histograms) and dynamic Bayesian networks, neither of which is applicable when the number of interacting objects is large. We treat the objects as point objects (referred to as "landmarks") and propose to model their changing configuration as a moving and deforming "shape" (using Kendall's shape theory for discrete landmarks). A continuous-state hidden Markov model is defined for landmark shape dynamics in an activity. The configuration of landmarks at a given time forms the observation vector, and the corresponding shape and the scaled Euclidean motion parameters form the hidden-state vector. An abnormal activity is then defined as a change in the shape activity model, which could be slow or drastic and whose parameters are unknown. Results are shown on a real abnormal activity-detection problem involving multiple moving objects.
Statistical shape theory for activity modeling

Monitoring activities in a certain region from video data is an important surveillance problem today. The goal is to learn the pattern of normal activities and detect unusual ones by identifying activities that deviate appreciably from the typical ones. In this paper we propose an approach using statistical shape theory (based on Kendall's shape model) [D.G. Kendall et al., 1999]. In a low resolution video each moving object is best represented as a moving point mass or particle. In this case, an activity can be defined by the interactions of all or some of these moving particles over time. We model this configuration of the particles by a polygonal shape formed from the locations of the points in a frame and the activity by the deformation of the polygons in time. These parameters are learnt for each typical activity. Given a test video sequence, an activity is classified as abnormal if the probability for the sequence (represented by the mean shape and the dynamics of the deviations), given the model is below a certain threshold. The approach gives very encouraging results in surveillance applications using a single camera and is able to identify various kinds of abnormal behaviors.
Information Weighted Consensus Filters and Their Application in Distributed Camera Networks

Due to their high fault-tolerance and scalability to large networks, consensus-based distributed algorithms have recently gained immense popularity in the sensor networks community. Large-scale camera networks are a special case. In a consensus-based state estimation framework, multiple neighboring nodes iteratively communicate with each other, exchanging their own local information about each target's state with the goal of converging to a single state estimate over the entire network. However, the state estimation problem becomes challenging when some nodes have limited observability of the state. In addition, the consensus estimate is suboptimal when the cross-covariances between the individual state estimates across different nodes are not incorporated in the distributed estimation framework. The cross-covariance is usually neglected because the computational and bandwidth requirements for its computation become unscalable for a large network. These limitations can be overcome by noting that, as the state estimates at different nodes converge, the information at each node becomes correlated. This fact can be utilized to compute the optimal estimate by proper weighting of the prior state and measurement information. Motivated by this idea, we propose information-weighted consensus algorithms for distributed maximum a posteriori parameter estimation, and their extension to the information-weighted consensus filter (ICF) for state estimation. We compare the performance of the ICF with existing consensus algorithms analytically, as well as experimentally by considering the scenario of a distributed camera network under various operating conditions.
Constrained optimization for opportunistic distributed visual sensing

Distributed networks of dynamically controllable pan-tilt-zoom (PTZ) cameras have high potential utility for tracking and high-res imaging of targets-of-interest maneuvering within a surveillance area. The actual utility that is achieved is determined by the real-time selection of the networked camera PTZ parameters to collaboratively achieve these objectives. This paper proposes a control mechanism for such a network to obtain opportunistic high-res facial imagery via distributed constrained optimization of PTZ parameters for each camera in the network. The objective function quantifies the per camera per target image quality. The tracking constraint that defines the feasible PTZ parameter space is a lower bound on the information about the estimated position for each target. All cameras optimize their PTZ parameters simultaneously using information broadcast by neighboring cameras. At certain time steps, due to the configuration of the targets relative to the cameras, and the fact that each camera may track many targets, the camera network may be able to achieve the tracking specification with remaining degrees-of-freedom that can be used to obtain high-res facial images from desirable aspect angles. The challenge is to define algorithms to automatically find these time instants, the appropriate imaging camera, and the appropriate parameter settings for all cameras to capitalize on these opportunities. The solution proposed herein involves a Bayesian formulation (for an automatic trade off of objective maximization versus the risk of losing track of any target), design of aligned local and global objective functions and the inequality constraint set, and development of a Distributed Lagrangian Consensus algorithm that allows cameras to exchange information and asymptotically converge on a pair of primal-dual optimal solutions. This article presents the theoretical solution along with simulation results.
Towards a criterion for evaluating the quality of 3D reconstructions

Even though numerous algorithms exist for estimating the structure of a scene from its video, the solutions obtained are often of unacceptable quality. To overcome some of the deficiencies, many application systems rely on processing more information than necessary with the hope that the redundancy will help improve the quality. This raises the question about how the accuracy of the solution is related to the amount of information processed by the algorithm. Can we define the accuracy of the solution precisely enough that we automatically recognize situations where the quality of the data is so bad that even a large number of additional observations will not yield the desired solution? This paper proposes an information theoretic criterion for evaluating the quality of a 3D reconstruction in terms of the statistics of the observed parameters (i.e. the image correspondences). The accuracy of the reconstruction is judged by considering the change in mutual information (or equivalently the conditional differential entropy) between a scene and its reconstructions and its effectiveness is shown through simulations.
A Physics-Based Analysis of Image Appearance Models

Linear and multilinear models (PCA, 3DMM, AAM/ASM, and multilinear tensors) of object shape/appearance have been very popular in computer vision. In this paper, we analyze the applicability of these heuristic models from the fundamental physical laws of object motion and image formation. We prove that under suitable conditions, the image appearance space can be closely approximated to be multilinear, with the illumination and texture subspaces being trilinearly combined with the direct sum of the motion and deformation subspaces. This result provides a physics-based understanding of many of the successes and limitations of the linear and multilinear approaches existing in the computer vision literature, and also identifies some of the conditions under which they are valid. It provides an analytical representation of the image space in terms of different physical factors that affect the image formation process. Numerical analysis of the accuracy of the physics-based models is performed, and tracking results on real data are presented.
Efficient cell segmentation and tracking of developing plant meristem

Analysis of Confocal Laser Scanning Microscopy (CLSM) images is gaining popularity in developmental biology for understanding growth dynamics. The automated analysis of such images is highly desirable for efficiency and accuracy. The first step in this process is segmentation and tracking leading to computation of cell lineages. In this paper, we present efficient, accurate, and robust segmentation and tracking algorithms for cells and detection of cell divisions in a 4D spatio-temporal image stack of a growing plant meristem. We show how to optimally choose the parameters in the watershed algorithm for high quality segmentation results. This yields high quality tracking results using cell correspondence evaluation functions. We show segmentation and tracking results on Confocal laser scanning microscopy data captured for 72 hours at every 3 hour intervals. Compared to recent results in this area, the proposed algorithms provide significantly longer cell lineages and more comprehensive identification of cell divisions.
Video classification based on social attitudes

Organizing large video databases is a pressing need and a challenging problem. Social attitudes in the form of users' beliefs and evaluations can benefit classification. For instance, news videos do not gather as much user attention as music videos while sports videos trigger interest mainly during the time of event. In this paper, we provide an extensive analysis of the role of usage statistics in aiding classification. Towards this, we propose a novel framework motivated by evolutionary biology to characterize growth, persistence and decline of contents in online environments. We then incorporate this information in a nearest neighbor classifier to establish categories. The effectiveness of the approach is demonstrated by comparing against results obtained using principal component analysis followed by nearest neighbor based classification.
Video summarization in a multi-view camera network

While most existing video summarization approaches aim to extract an informative summary of a single video, we propose a novel framework for summarizing multi-view videos by exploiting both intra- and inter-view content correlations in a joint embedding space. We learn the embedding by minimizing an objective function that has two terms: one due to intra-view correlations and another due to inter-view correlations across the multiple views. The solution can be obtained directly by solving one Eigen-value problem that is linear in the number of multi-view videos. We then employ a sparse representative selection approach over the learned embedding space to summarize the multi-view videos. Experimental results on several benchmark datasets demonstrate that our proposed approach clearly out-performs the state-of-the-art.
Collaborative Sensing in a Distributed PTZ Camera Network

The performance of dynamic scene algorithms often suffers because of the inability to effectively acquire features on the targets, particularly when they are distributed over a wide field of view. In this paper, we propose an integrated analysis and control framework for a pan, tilt, zoom (PTZ) camera network in order to maximize various scene understanding performance criteria (e.g., tracking accuracy, best shot, and image resolution) through dynamic camera-to-target assignment and efficient feature acquisition. Moreover, we consider the situation where processing is distributed across the network since it is often unrealistic to have all the image data at a central location. In such situations, the cameras, although autonomous, must collaborate among themselves because each camera's PTZ parameter entails constraints on the others. Motivated by recent work in cooperative control of sensor networks, we propose a distributed optimization strategy, which can be modeled as a game involving the cameras and targets. The cameras gain by reducing the error covariance of the tracked targets or through higher resolution feature acquisition, which, however, comes at the risk of losing the dynamic target. Through the optimization of this reward-versus-risk tradeoff, we are able to control the PTZ parameters of the cameras and assign them to targets dynamically. The tracks, upon which the control algorithm is dependent, are obtained through a consensus estimation algorithm whereby cameras can arrive at a consensus on the state of each target through a negotiation strategy. We analyze the performance of this collaborative sensing strategy in active camera networks in a simulation environment, as well as a real-life camera network.
Opportunistic Image Acquisition of Individual and Group Activities in a Distributed Camera Network

The decreasing cost and size of video sensors has led to camera networks becoming pervasive in our lives. However, the ability to analyze these images effectively is very much a function of the quality of the acquired images. In this paper, we consider the problem of automatically controlling the fields of view of individual pan–tilt–zoom (PTZ) cameras in a camera network leading to improved situation awareness (e.g., where and what are the critical targets and events) in a region of interest. The network of cameras attempts to observe the entire region of interest at some minimum resolution while opportunistically acquiring high resolution images of critical events in real time. Since many activities involve groups of people interacting, an important decision that the network needs to make is whether to focus on individuals or groups of them. This is achieved by understanding the performance of video analysis tasks and designing camera control strategies to improve a metric that quantifies the quality of the source imagery. Optimization strategies, along with a distributed implementation, are proposed, and their theoretical properties analyzed. The proposed methods bring together computer vision and network control ideas. The performance of the proposed methodologies discussed herein has been evaluated on a real-life wireless network of PTZ capable cameras.
Statistical shape theory for activity modeling

Monitoring activities in a certain region from video data is an important surveillance problem. The goal is to learn the pattern of normal activities and detect unusual ones by identifying activities that deviate appreciably from the typical ones. We propose an approach using statistical shape theory based on the shape model of D.G. Kendall et al. (see "Shape and Shape Theory", John Wiley and Sons, 1999). In a low resolution video, each moving object is best represented as a moving point mass or particle. In this case, an activity can be defined by the interactions of all or some of these moving particles over time. We model this configuration of the particles by a polygonal shape formed from the locations of the points in a frame and the activity by the deformation of the polygons in time. These parameters are learned for each typical activity. Given a test video sequence, an activity is classified as abnormal if the probability for the sequence (represented by the mean shape and the dynamics of the deviations), given the model, is below a certain threshold The approach gives very encouraging results in surveillance applications using a single camera and is able to identify various kinds of abnormal behavior.
Automated registration of live imaging stacks of Arabidopsis

For actively developing tissues, a computational platform capable of automatically registering, segmenting and tracking cells is very critical to obtaining high-throughput and quantitative measurements of a range of cell behaviors, and can lead to a better understanding of the underlying dynamics of morphogenesis. In this work, we present an automated landmark-based registration method to register shoot apical meristem of Arabidopsis Thaliana images obtained through the Confocal Laser Scanning Microscopy technique. The proposed landmark-based registration method uses local graph-based approach to automatically find corresponding landmark pairs. The registration algorithm combined with an existing tracking method is tested on multiple datasets and it significantly improves the accuracy of cell lineages and division statistics.