nils-herrmann/clean_scirepeval huggingface dataset:
Learning to move autonomously in a hostile world

This sequence was recorded from a live, interactive demo, in which the user controls the crate and can move it anywhere at anytime. A virtual agent is tasked with traveling from the left of the scene to the target on the right. While the agent is running towards the target, the user moves the crate into the position shown on the left, blocking the agent’s intended path. The local motion planner selects frames that avoid hitting the object but still make progress toward the goal (A). The user again moves the object into the agent’s path, and again the system successfully copes (B). Altogether, the user tries to block the agent by moving the crate 3 times, but the agent still dodges it and arrives at the target position seamlessly.
Strike a pose: tracking people by finding stylized poses

We develop an algorithm for finding and kinematically tracking multiple people in long sequences. Our basic assumption is that people tend to take on certain canonical poses, even when performing unusual activities like throwing a baseball or figure skating. We build a person detector that quite accurately detects and localizes limbs of people in lateral walking poses. We use the estimated limbs from a detection to build a discriminative appearance model; we assume the features that discriminate a figure in one frame will discriminate the figure in other frames. We then use the models as limb detectors in a pictorial structure framework, detecting figures in unrestricted poses in both previous and successive frames. We have run our tracker on hundreds of thousands of frames, and present and apply a methodology for evaluating tracking on such a large scale. We test our tracker on real sequences including a feature-length film, an hour of footage from a public park, and various sports sequences. We find that we can quite accurately automatically find and track multiple people interacting with each other while performing fast and unusual motions.
Skeletal Parameter Estimation from Optical Motion Capture Data

In this paper we present an algorithm for automatically estimating a subject's skeletal structure from optical motion capture data. Our algorithm consists of a series of steps that cluster markers into segment groups, determine the topological connectivity between these groups, and locate the positions of their connecting joints. Our problem formulation makes use of fundamental distance constraints that must hold for markers attached to an articulated structure, and we solve the resulting systems using a combination of spectral clustering and nonlinear optimization. We have tested our algorithms using data from both passive and active optical motion capture devices. Our results show that the system works reliably even with as few as one or two markers on each segment. For data recorded from human subjects, the system determines the correct topology and qualitatively accurate structure. Tests with a mechanical calibration linkage demonstrate errors for inferred segment lengths on average of only two percent. We discuss applications of our methods for commercial human figure animation, and for identifying human or animal subjects based on their motion independent of marker placement or feature selection.
Tracking People and Recognizing Their Activities

We present a system for automatic people tracking and activity recognition. Our basic approach to people-tracking is to build an appearance model for the person in the video. The video illustrates our method of using a stylized-pose detector. Our system builds a model of limb appearance from those sparse stylized detections. Our algorithm then reprocesses the video, using the learned appearance models to find people in unrestricted configuration. We can use our tracker to recover 3D configurations and activity labels. We assume we have a motion capture library where the 3D poses have been labeled offline with activity descriptions.
Skeletal parameter estimation from optical motion capture data

In this paper we present an algorithm for automatically estimating a subject's skeletal structure from optical motion capture data. Our algorithm consists of a series of steps that cluster markers into segment groups, determine the topological connectivity between these groups, and locate the positions of their connecting joints. Our problem formulation makes use of fundamental distance constraints that must hold for markers attached to an articulated structure, and we solve the resulting systems using a combination of spectral clustering and nonlinear optimization. We have tested our algorithms using data from both passive and active optical motion capture devices. Our results show that the system works reliably even with as few as one or two markers on each segment. For data recorded from human subjects, the system determines the correct topology and qualitatively accurate structure. Tests with a mechanical calibration linkage demonstrate errors for inferred segment lengths on average of only two percent. We discuss applications of our methods for commercial human figure animation, and for identifying human or animal subjects based on their motion independent of marker placement or feature selection.
Looking at People

There is a great need for programs that can describe what people are doing from video. This is difficult to do, because it is hard to identify and track people in video sequences, because we have no canonical vocabulary for describing what people are doing, and because the interpretation of what people are doing depends very strongly on what is nearby. Tracking is hard, because it is important to track relatively small structures that can move relatively fast for example, lower arms. I will describe research into kinematic tracking tracking that reports the kinematic configuration of the body that has resulted in a fairly accurate, fully automatic tracker, that can keep track of multiple people. Once one has tracked the body, one must interpret the results. One way to do so is to have a motion synthesis system that takes the track, and produces a motion that is (a) like a human motion and (b) close to the track. Our work has produced a high-quality motion synthesis system that can produce motions that look very much like human activities. I will describe work that couples that system with a tracker to produce a description of the activities, entirely automatically. I will speculate on some of the many open problems. What should one report? How do nearby objects affect one’s interpretation of activities? How can one interpret patterns of behavior?
Cloth capture

We present a method for capturing the geometry and parameterization of fastmoving cloth using multiple video cameras, without requiring camera calibration. Our cloth is printed with a multiscale pattern that allows capture at both high speed and high spatial resolution even though self-occlusion might block any individual camera from seeing the majority of the cloth. We show how to incorporate knowledge of this pattern into conventional structure from motion approaches, and use a novel scheme for camera calibration using the pattern, derived from the shape from texture literature. By combining strain minimization with the point reconstruction we produce visually appealing cloth sequences. We demonstrate our algorithm by capturing, retexturing and displaying several sequences of fast
Learning the Behavior of Users in a Public Space through Video Tracking

The paper describes a video tracking system that tracks and analyzes the behavioral pattern of users in a public space. We have obtained important statistical measurements about users' behavior, which can be used to evaluate architectural design in terms of human spatial behavior and model the behavior of users in public spaces. Previously, such measurements could only be obtained through costly manual processes, e.g. behavioral mapping and time-lapse filming with human examiners. Our system has automated the process of analyzing the behavior of users. The system consists of a head detector for detecting people in each single frame of the video and data association for tracking people through frames. We compared the results obtained using our system with those obtained by manual counting, for a small data set, and found the results to be fairly accurate. We then applied the system to a large-scale data set and obtained substantial statistical measurements of parameters such as the total number of users who entered the space, the total number of users who sat by a fountain, the time that each spent by the fountain, etc. These statistics allow fundamental rethinking of the way people use a public space. This research is a novel application of computer vision in evaluating architectural design in terms of human behavior
Efficient Unsupervised Learning for Localization and Detection in Object Categories

We describe a novel method for learning templates for recognition and localization of objects drawn from categories. A generative model represents the configuration of multiple object parts with respect to an object coordinate system; these parts in turn generate image features. The complexity of the model in the number of features is low, meaning our model is much more efficient to train than comparative methods. Moreover, a variational approximation is introduced that allows learning to be orders of magnitude faster than previous approaches while incorporating many more features. This results in both accuracy and localization improvements. Our model has been carefully tested on standard datasets; we compare with a number of recent template models. In particular, we demonstrate state-of-the-art results for detection and localization.
Deforming Objects Provide Better Camera Calibration

Long-term monitoring of pollutants or moisture in a fluid stream, such as an air stream, is achieved by electrically measuring (and optionally recording) pressure changes caused by a reaction between a pollutant or moisture in the stream and a chemical capable of reacting with such pollutant or moisture. The invention is useful for monitoring apparatus used in environment protection as well as measuring apparatus in determining moisture concentrations in, for example, protective or carrier gas streams utilized in semiconductor technology.
Computational Studies of Human Motion: Part 1, Tracking and Motion Synthesis

We review methods for kinematic tracking of the human body in video. The review is part of a projected book that is intended to cross-fertilize ideas about motion representation between the animation and computer vision communities. The review confines itself to the earlier stages of motion, focusing on tracking and motion synthesis; future material will cover activity representation and motion generation. In general, we take the position that tracking does not necessarily involve (as is usually thought) complex multimodal inference problems. Instead, there are two key problems, both easy to state. The first is lifting, where one must infer the configuration of the body in three dimensions from image data. Ambiguities in lifting can result in multimodal inference problem, and we review what little is known about the extent to which a lift is ambiguous. The second is data association, where one must determine which pixels in an image Full text available at: http://dx.doi.org/10.1561/0600000005
Motion Analysis by Synthesis: Automatically Annotating Activities in Video

An important vision problem is to automatically describe what people are doing a sequence of video. This problem is difficult for many reasons. First, tracking the articulations of people is difficult because arms and legs are small and can move quite fast. Secondly, one must sew the estimated poses from each frame together into a coherent motion path; this is difficult because people can move in unpredictable ways. Finally, one must describe the motion with some activity annotation. This appears difficult for everyday motion since there may not be a canonical vocabulary of motions. We describe a fully automatic system that labels the activities of multiple people in a video sequence. The system decouples the choice of annotation vocabulary from the analysis procedure, allowing for easy revision of the vocabulary. The system first uses a pictorial structure model to independently detect 2D poses in each frame. The system then synthesizes 3D motion clips that looks like the 2D motions by matching poses to a stored library of motion capture data. The motion capture data is labeled offline with a class structure that allows for multiple annotations to be composed; one may walk, while waveing, for example. The lack of a canonical vocabulary also makes it difficult to evaluate experimental results. We introduce a mutual information criterion that allows one to evaluate different annotation systems given labeled test footage. We demonstrate and evaluate our system on real sequences of multiple people interacting and commercial shots from a feature-length film.
Pushing people around

We present an algorithm for animating characters being pushed by an external source such as a user or a game environment. We start with a collection of motions of a real person responding to being pushed. When a character is pushed, we synthesize new motions by picking a motion from the recorded collection and modifying it so that the character responds to the push from the desired direction and location on its body. Determining the deformation parameters that realistically modify a recorded response motion is difficult. Choosing the response motion that will look best when modified is also non-trivial, especially in real-time. To estimate the envelope of deformation parameters that yield visually plausible modifications of a given motion, and to find the best motion to modify, we introduce an oracle. The oracle is trained using a set of synthesized response motions that are identified by a user as good and bad. Once trained, the oracle can, in real-time, estimate the visual quality of all motions in the collection and required deformation parameters to serve a desired push.Our method performs better than a baseline algorithm of picking the closest response motion in configuration space, because our method can find visually plausible transitions that do not necessarily correspond to similar motions in terms of configuration. Our method can also start with a limited set of recorded motions and modify them so that they can be used to serve different pushes on the upper body.
Fast and detailed approximate global illumination by irradiance decomposition

In this paper we present an approximate method for accelerated computation of the final gathering step in a global illumination algorithm. Our method operates by decomposing the radiance field close to surfaces into separate far- and near-field components that can be approximated individually. By computing surface shading using these approximations, instead of directly querying the global illumination solution, we have been able to obtain rendering time speed ups on the order of 10x compared to previous acceleration methods. Our approximation schemes rely mainly on the assumptions that radiance due to distant objects will exhibit low spatial and angular variation, and that the visibility between a surface and nearby surfaces can be reasonably predicted by simple location and orientation-based heuristics. Motivated by these assumptions, our far-field scheme uses scattered-data interpolation with spherical harmonics to represent spatial and angular variation, and our near-field scheme employs an aggressively simple visibility heuristic. For our test scenes, the errors introduced when our assumptions fail do not result in visually objectionable artifacts or easily noticeable deviation from a ground-truth solution. We also discuss how our near-field approximation can be used with standard local illumination algorithms to produce significantly improved images at only negligible additional cost.
Planar object recognition using projective shape representation

We describe a model based recognition system, called LEWIS, for the identification of planar objects based on a projectively invariant representation of shape. The advantages of this shape description include simple model acquisition (direct from images), no need for camera calibration or object pose computation, and the use of index functions. We describe the feature construction and recognition algorithms in detail and provide an analysis of the combinatorial advantages of using index functions. Index functions are used to select models from a model base and are constructed from projective invariants based on algebraic curves and a canonical projective coordinate frame. Examples are given of object recognition from images of real scenes, with extensive object libraries. Successful recognition is demonstrated despite partial occlusion by unmodelled objects, and realistic lighting conditions.
Recovering shape and irradiance maps from rich dense texton fields

We describe a method that recovers an estimate of surface shape and of the irradiance field for a textured surface. The method assumes the surface is viewed in scaled orthography, and we demonstrate the appropriateness of this assumption. Our method uses interest points to obtain the locations of putative texton instances, clusters the textons into types, and then uses an autocalibration method to recover the frontal appearance of each texton model. This yields (a) a dense set of normal estimates, each up to a two-fold ambiguity, (b) a dense set of irradiance estimates and (c) whether each instance is, in fact, an instance of the relevant texton. Because we are able to obtain a very large number of instances of a large number of different textons, this information is obtained at sites very closely spaced in the image. As a result, we need only a simple smoothness constraint to reconstruct a surface model, using EM to resolve the normal ambiguity. We show results on images of real scenes, comparing our reconstructions with those obtained using other methods and demonstrating the accuracy of both the recovered shape and the irradiance estimate.
Making Latin Manuscripts Searchable using gHMMs

We describe a method that can make a scanned, handwritten mediaeval latin manuscript accessible to full text search. A generalized HMM is fitted, using transcribed latin to obtain a transition model and one example each of 22 letters to obtain an emission model. We show results for unigram, bigram and trigram models. Our method transcribes 25 pages of a manuscript of Terence with fair accuracy (75% of letters correctly transcribed). Search results are very strong; we use examples of variant spellings to demonstrate that the search respects the ink of the document. Furthermore, our model produces fair searches on a document from which we obtained no training data.
Radiance caching and local geometry correction

We present a final gather algorithm which splits the irradiance integral into two components. One component captures the incident radiance due to distant surfaces. This incident radiance due to far field illumination is represented as a spatially varying field of spherical harmonic coefficients. Since distant surfaces do not cause rapid changes in incident radiance, this field is smooth and slowly varying and can be computed quickly and represented efficiently.In contrast, nearby surfaces may create drastic changes in irradiance, because their positions on the visible hemisphere can change quickly. We can find such nearby surfaces (scene triangles) by a local search. By assuming nearby surfaces are always visible, we can correct the far field irradiance estimate we obtain using the spherical harmonics, and restore the high frequency detail in indirect lighting. This correction can be performed efficiently because finding nearby surfaces is a local operation.
Probabilistic Methods for Finding People

Finding people in pictures presents a particularly difficult object recognition problem. We show how to find people by finding candidate body segments, and then constructing assemblies of segments that are consistent with the constraints on the appearance of a person that result from kinematic properties. Since a reasonable model of a person requires at least nine segments, it is not possible to inspect every group, due to the huge combinatorial complexity.We propose two approaches to this problem. In one, the search can be pruned by using projected versions of a classifier that accepts groups corresponding to people. We describe an efficient projection algorithm for one popular classifier, and demonstrate that our approach can be used to determine whether images of real scenes contain people.The second approach employs a probabilistic framework, so that we can draw samples of assemblies, with probabilities proportional to their likelihood, which allows to draw human-like assemblies more often than the non-person ones. The main performance problem is in segmentation of images, but the overall results of both approaches on real images of people are encouraging.
Towards auto-documentary: tracking the evolution of news stories

News videos constitute an important source of information for tracking and documenting important events. In these videos, news stories are often accompanied by short video shots that tend to be repeated during the course of the event. Automatic detection of such repetitions is essential for creating auto-documentaries, for alleviating the limitation of traditional textual topic detection methods. In this paper, we propose novel methods for detecting and tracking the evolution of news over time. The proposed method exploits both visual cues and textual information to summarize evolving news stories. Experiments are carried on the TREC-VID data set consisting of 120 hours of news videos from two different channels.
Who's In the Picture

The context in which a name appears in a caption provides powerful cues as to who is depicted in the associated image. We obtain 44,773 face images, using a face detector, from approximately half a million captioned news images and automatically link names, obtained using a named entity recognizer, with these faces. A simple clustering method can produce fair results. We improve these results significantly by combining the clustering process with a model of the probability that an individual is depicted given its context. Once the labeling procedure is over, we have an accurately labeled set of faces, an appearance model for each individual depicted, and a natural language model that can produce accurate results on captions in isolation.
Enriching a motion collection by transplanting limbs

This paper describes a method that can significantly increase the size of a collection of motion observations by cutting limbs from one motion sequence and attaching them to another. Not all such transplants are successful, because correlations across the body are a significant feature of human motion. The method uses randomized search based around a set of rules to generate transplants that are (a) likely to be successful and (b) likely to enrich the existing motion collection. The resulting frames are annotated by a classifier to tell whether they look like human motion or not.
 We evaluate the method by obtaining motion demands from an application, synthesizing motions to meet those demands, and then scoring the synthesized motions. Motions synthesized using transplants are generally somewhat better than those synthesized without using transplants, because transplanting generates many frames quite close to the original frames, so that it is easier for the motion synthesis process to find a good path in the motion graph. Furthermore, we show classifier errors tend to have relatively little impact in practice.
 Finally, we show that transplanted motion data can be used to synthesize motions of a group coordinated in space and time without producing motions that share frames.