nils-herrmann/clean_scirepeval huggingface dataset:
Euclidean Reconstruction and Reprojection Up to Subgroups

The necessary and sufficient conditions for being able to estimate scene structure, motion and camera calibration from a sequence of images are very rarely satisfied in practice. What exactly can be estimated in sequences of practical importance, when such conditions are not satisfied? In this paper we give a complete answer to this question. For every camera motion that fails to meet the conditions, we give explicit formulas for the ambiguities in the reconstructed scene, motion and calibration. Such a characterization is crucial both for designing robust estimation algorithms (that do not try to recover parameters that cannot be recovered), and for generating novel views of the scene by controlling the vantage point. To this end, we characterize explicitly all the vantage points that give rise to a valid Euclidean reprojection regardless of the ambiguity in the reconstruction. We also characterize vantage points that generate views that are altogether invariant to the ambiguity. All the results are presented using simple notation that involves no tensors nor complex projective geometry, and should be accessible with basic background in linear algebra.
Shedding light on stereoscopic segmentation

We propose a variational algorithm to jointly estimate the shape, albedo, and light configuration of a Lambertian scene from a collection of images taken from different vantage points. Our work can be thought of as extending classical multi-view stereo to cases where point correspondence cannot be established, or extending classical shape from shading to the case of multiple views with unknown light sources. We show that a first naive formalization of this problem yields algorithms that are numerically unstable, no matter how close the initialization is to the true geometry. We then propose a computational scheme to overcome this problem, resulting in provably stable algorithms that converge to (local) minima of the cost functional. Although we restrict our attention to Lambertian objects with uniform albedo, extensions of our framework are conceivable.
Title Region-based segmentation on evolving surfaces with application to 3 D shape and radiance estimation Permalink

We consider the problem of estimating the shape and radiance of a scene from a calibrated set of images under the assumption that the scene is Lambertian and its radiance is piecewise constant. We model the radiance segmentation explicitly using smooth curves on the surface that bound regions of constant radiance. We pose the scene reconstruction problem in a variational framework, where the unknowns are the surface, the radiance values and the segmenting curves. We propose an iterative procedure to minimize a global cost functional that combines geometric priors on both the surface and the curves with a data fitness score. We carry out the numerical implementation in the level set framework.
Stereoscopic Segmentation

We cast the problem of multiframe stereo reconstruction of a smooth shape as the global region segmentation of a collection of images of the scene. Dually, the problem of segmenting multiple calibrated images of an object becomes that of estimating the solid shape that gives rise to such images. We assume that the radiance of the scene results in piecewise homogeneous image statistics. This simplifying assumption covers Lambertian scenes with constant albedo as well as fine homogeneous textures, which are known challenges to stereo algorithms based on local correspondence. We pose the segmentation problem within a variational framework, and use fast level set methods to find the optimal solution numerically. Our algorithm does not work in the presence of strong photometric features, where traditional reconstruction algorithms do. It enjoys significant robustness to noise under the assumptions it is designed for.
Spatially Homogeneous Dynamic Textures

We address the problem of modeling the spatial and temporal second-order statistics of video sequences that exhibit both spatial and temporal regularity, intended in a statistical sense. We model such sequences as dynamic multiscale autoregressive models, and introduce an efficient algorithm to learn the model parameters. We then show how the model can be used to synthesize novel sequences that extend the original ones in both space and time, and illustrate the power, and limitations, of the models we propose with a number of real image sequences.
Region-based segmenta-tion on evolving surfaces with application to 3D shape and radiance estimation

Region-based Segmentation on Evolving Surfaces with Application to 3D Reconstruction of Shape and Piecewise Constant Radiance Hailin Jin 1 , Anthony J. Yezzi 2 , and Stefano Soatto 1 Computer Science Department, University of California, Los Angeles, CA 90095, hljin,soatto@cs.ucla.edu School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332,  We consider the problem of estimating the shape and radi- ance of a scene from a calibrated set of images under the assumption that the scene is Lambertian and its radiance is piecewise constant. We model the radiance segmentation explicitly using smooth curves on the surface that bound regions of constant radiance. We pose the scene re- construction problem in a variational framework, where the unknowns are the surface, the radiance values and the segmenting curves. We pro- pose an iterative procedure to minimize a global cost functional that combines geometric priors on both the surface and the curves with a data ﬁtness score. We carry out the numerical implementation in the level set framework.  variational methods, Mumford-Shah functional, image segmen- tation, multi-view stereo, level set methods, curve evolution on manifolds.  A plush model of “nemo.” The object exhibits piecewise constant appearance. From a set of calibrated views, our algorithm can estimate the shape and the piecewise constant radiance.  Inferring three-dimensional shape and appearance of a scene from a collection of images has been a central problem in Computer Vision, known as multi- view stereo. Traditional approaches to this problem ﬁrst match points or small
Tracking deformable moving objects under severe occlusions

We propose a nonlinear model for tracking a slowly deforming and moving contour despite significant occlusions. The contour is represented implicitly, and its motion is described by the action of a finite-dimensional group; we estimate both the implicit representation of the contour (its shape) and its motion. Our contribution consists in defining a generative model that is not subject to arbitrary re-parameterization, choice of (non-unique) key points or control points, and allows enforcing a dynamical model of motion when it is available. Otherwise, our approach allows enforcing simple phenomenological models, for instance low acceleration or low jerk.
Observing Shape from Defocused Images

Accommodation cues are measurable properties of an image that are associated with a change in the geometry of the imaging device. To what extent can three-dimensional shape be reconstructed using accommodation cues alone? This question is fundamental to the problem of reconstructing shape from focus (SFF) and shape from defocus (SFD) for applications in inspection, microscopy, image restoration and visualization. We address it by studying the “observability” of accommodation cues in an analytical framework that reveals under what conditions shape can be reconstructed from defocused images. We do so in three steps: (1) we characterize the observability of any surface in the presence of a controlled radiance (“weak observability”), (2) we conjecture the existence of a radiance that allows distinguishing any two surfaces (“sufficient excitation”) and (3) we show that in the absence of any prior knowledge on the radiance, two surfaces can be distinguished up to the degree of resolution determined by the complexity of the radiance (“strong observability”). We formulate the problem of reconstructing the shape and radiance of a scene as the minimization of the information divergence between blurred images, and propose an algorithm that is provably convergent and guarantees that the solution is admissible, in the sense of corresponding to a positive radiance and imaging kernel.
Deformotion: Deforming Motion, Shape Average and the Joint Registration and Approximation of Structures in Images

What does it mean for a deforming object to be “moving”? How can we separate the overall motion (a finite-dimensional group action) from the more general deformation (a diffeomorphism)? In this paper we propose a definition of motion for a deforming object and introduce a notion of “shape average” as the entity that separates the motion from the deformation. Our definition allows us to derive novel and efficient algorithms to register non-identical shapes using region-based methods, and to simultaneously approximate and align structures in greyscale images. We also extend the notion of shape average to that of a “moving average” in order to track moving and deforming objects through time. The algorithms we propose extend prior work on landmark-based matching to smooth curves, and involve the numerical integration of partial differential equations, which we address within the framework of level set methods.
Optimal Structure from Motion: Local Ambiguities and Global Estimates

From Motion” (SFM) refers to the problem of estimating spatial properties of a three-dimensional scene from the motion of its projection onto a two-dimensional surface, such as the retina. We present an analysis of SFM which results in algorithms that are provably convergent and provably optimal with respect to a chosen norm.In particular, we cast SFM as the minimization of a high-dimensional quadratic cost function, and show how it is possible to reduce it to the minimization of a two-dimensional function whose stationary points are in one-to-one correspondence with those of the original cost function. As a consequence, we can plot the reduced cost function and characterize the configurations of structure and motion that result in local minima. As an example, we discuss two local minima that are associated with well-known visual illusions. Knowledge of the topology of the residual in the presence of such local minima allows us to formulate minimization algorithms that, in addition to provably converge to stationary points of the original cost function, can switch between different local extrema in order to converge to the global minimum, under suitable conditions. We also offer an experimental study of the distribution of the estimation error in the presence of noise in the measurements, and characterize the sensitivity of the algorithm using the structure of Fisher's Information matrix.
Modeling and Synthesis of Facial Motion Driven by Speech

We introduce a novel approach to modeling the dynamics of human facial motion induced by the action of speech for the purpose of synthesis. We represent the trajectories of a number of salient features on the human face as the output of a dynamical system made up of two subsystems, one driven by the deterministic speech input, and a second driven by an unknown stochastic input. Inference of the model (learning) is performed automatically and involves an extension of independent component analysis to time-depentend data. Using a shape-texture decompositional representation for the face, we generate facial image sequences reconstructed from synthesized feature point positions.
Image Primitives and Correspondence

In previous chapters we have seen how geometric primitives, such as points and lines in space, can be transformed so that one can compute the coordinates of their “image,” i.e. their projection onto the image plane. In practice, however, images are arrays of positive numbers that measure the amount of light incident on a sensor at a particular location (see Sections 3.1 and 3.2, and Appendix 3.A). So, how do we reconcile a geometric image formation model (Section 3.3) with the fact that what we measure with a camera is not points and lines, but light intensity? In other words, how do we go from measurements of light (photometry) to geometry? This is the subject of this chapter: we will show how geometric primitives can be extracted from photometric measurements and matched across different views, so that the rest of the book can concentrate on geometry.
Scene and Motion Reconstruction from Defocused and Motion-Blurred Images via Anisotropic Diffusion

We propose a solution to the problem of inferring the depth map, radiance and motion of a scene from a collection of motion-blurred and defocused images. We model motion-blur and defocus as an anisotropic diffusion process, whose initial conditions depend on the radiance and whose diffusion tensor encodes the shape of the scene, the motion field and the optics parameters. We show that this model is well-posed and propose an efficient algorithm to infer the unknowns of the model. Inference is performed by minimizing the discrepancy between the measured blurred images and the ones synthesized via forward diffusion. Since the problem is ill-posed, we also introduce additional Tikhonov regularization terms. The resulting method is fast and robust to noise as shown by experiments with both synthetic and real data.
A variational approach to scene reconstruction and image segmentation from motion-blur cues

In this paper we are interested in the joint reconstruction of geometry and photometry of scenes with multiple moving objects from a collection of motion-blurred images. We make simplifying assumptions on the photometry of the scene (we neglect complex illumination effects) and infer the motion field of the scene, its depth map, and its radiance. In particular, we choose to partition the image into regions where motion is well approximated by a simple planar translation. We model motion-blurred images as the solution of an anisotropic diffusion equation, whose initial conditions depend on the radiance and whose diffusion tensor encodes the depth map of the scene and the motion field. We propose an algorithm to infer the unknowns of the model by minimizing the discrepancy between the measured images and the ones synthesized via diffusion. Since the problem is ill-posed, we also introduce additional Tikhonov regularization terms.
Representation of a Three-Dimensional Moving Scene

The study of the geometric relationship between a three-dimensional (3-D) scene and its two-dimensional (2-D) images taken from a moving camera is at heart the interplay between two fundamental sets of transformations: Euclidean motion, also called rigid-body motion,which models how the camera moves, and perspective projection,which describes the image formation process. Long before these two transformations were brought together in computer vision, their theory had been developed independently. The study of the principles of motion of a material body has a long history belonging to the foundations of mechanics. For our purpose, more recent noteworthy insights to the understanding of the motion of rigid objects came from Chasles and Poinsot in the early 1800s. Their findings led to the current treatment of this subject, which has since been widely adopted.
Multiple-View Geometry of Points and Lines

In this chapter we study how the framework of epipolar geometry introduced in Part II generalizes to the case of multiple views. As we shall see, this entails studying constraints that corresponding points in different views must satisfy if they are the projection of the same point in space. Not only is this development crucial for understanding the geometry of multiple views but, as in the two-view case, these constraints may be used to derive algorithms for reconstructing camera configuration and, ultimately, the 3-D position of geometric primitives. The search for the m-view analogue of the epipolar constraint has been an active research area for almost two decades. It was realized early on in [Liu and Huang, 1986, Spetsakis and Aloimonos, 1987] that the relationship between three views of the same point or line can be characterized by the trilinear constraints. Consequently, the study of multiple-view geometry has involved multidimensional linear operators, also called tensors.
Fast Visual Feature Selection and Tracking in a Hybrid Reconfigurable Architecture

We present a fast visual feature tracking system which takes advantage of dedicated hardware to perform the computationally intensive step of selection. A software system uses the output of the hardware selector to develop tracks using filtering and data association techniques, and image-based validation. By using separate, custom hardware to select hundreds of points per frame then associating these points to tracks in software, we achieve greater than 60Hz real-time tracking.
Reconstruction from Two Calibrated Views

In this chapter we begin unveiling the basic geometry that relates images of points to their 3-D position. We start with the simplest case of two calibrated cameras, and describe an algorithm, first proposed by the British psychologist H.C. Longuet-Higgins in 1981, to reconstruct the relative pose (i.e. position and orientation) of the cameras as well as the locations of the points in space from their projection onto the two images.
Integral Invariant Signatures

For shapes represented as closed planar contours, we introduce a class of functionals that are invariant with respect to the Euclidean and similarity group, obtained by performing integral operations. While such integral invariants enjoy some of the desirable properties of their differential cousins, such as locality of computation (which allows matching under occlusions) and uniqueness of representation (in the limit), they are not as sensitive to noise in the data. We exploit the integral invariants to define a unique signature, from which the original shape can be reconstructed uniquely up to the symmetry group, and a notion of scale-space that allows analysis at multiple levels of resolution. The invariant signature can be used as a basis to define various notions of distance between shapes, and we illustrate the potential of the integral invariant representation for shape matching on real and synthetic data.
Region-Based Segmentation on Evolving Surfaces with Application to 3D Reconstruction of Shape and Piecewise Constant Radiance

We consider the problem of estimating the shape and radiance of a scene from a calibrated set of images under the assumption that the scene is Lambertian and its radiance is piecewise constant. We model the radiance segmentation explicitly using smooth curves on the surface that bound regions of constant radiance. We pose the scene reconstruction problem in a variational framework, where the unknowns are the surface, the radiance values and the segmenting curves. We propose an iterative procedure to minimize a global cost functional that combines geometric priors on both the surface and the curves with a data fitness score. We carry out the numerical implementation in the level set framework.
Kernel Density Estimation and Intrinsic Alignment for Knowledge-Driven Segmentation: Teaching Level Sets to Walk

We address the problem of image segmentation with statistical shape priors in the context of the level set framework. Our paper makes two contributions: Firstly, we propose a novel multi-modal statistical shape prior which allows to encode multiple fairly distinct training shapes. This prior is based on an extension of classical kernel density estimators to the level set domain. Secondly, we propose an intrinsic registration of the evolving level set function which induces an invariance of the proposed shape energy with respect to translation. We demonstrate the advantages of this multi-modal shape prior applied to the segmentation and tracking of a partially occluded walking person.
Dynamic Textures

Dynamic textures are sequences of images of moving scenes that exhibit certain stationarity properties in time; these include sea-waves, smoke, foliage, whirlwind etc. We present a characterization of dynamic textures that poses the problems of modeling, learning, recognizing and synthesizing dynamic textures on a firm analytical footing. We borrow tools from system identification to capture the “essence” of dynamic textures; we do so by learning (i.e. identifying) models that are optimal in the sense of maximum likelihood or minimum prediction error variance. For the special case of second-order stationary processes, we identify the model sub-optimally in closed-form. Once learned, a model has predictive power and can be used for extrapolating synthetic sequences to infinite length with negligible computational cost. We present experimental evidence that, within our framework, even low-dimensional models can capture very complex visual phenomena.
Extension to General Incidence Relations

This chapter1 extends development in the previous chapter to the study of all incidence relations among different geometric primitives in 3-D space and in multiple images (e.g., intersection and coplanarity). We will demonstrate how incidence relations among multiple points, lines, and planes in space can be encoded in multiple images through the same matrix rank conditions. Such a generalization reveals additional instances that give rise to some nontrivial constraints among features in multiple views. This revelation will in turn lead to a more general class of techniques for structure and motion recovery that can use a multitude of geometric features simultaneously and exploit arbitrary incidence relations among them.
Multiple-view geometry for image-based modeling

This course presents the state of the art in multiple-view geometry, including methods and algorithms for reconstructing 3-D geometric models of scenes from video or photographs. This course is based on a novel approach to multiple-view geometry that only requires linear algebra, as opposed to more involved projective and algebraic geometry that most current methods employ. This new approach aims to make image-based modeling techniques accessible to a larger audience compared to existing ones. The presentations will be based on a recent Springer-Verlag textbook on 3D vision (coauthored by the course organizers). Matlab code for the algorithms will be made available at the book website.
Gait recognition using dynamic affine invariants

We present a method for recognizing classes of human gaits from video sequences. We propose a novel image based representation for human gaits. At any instance of time a gait is represented by a vector of affine invariant moments. The invariants are computed on the binary silhouettes corresponding to the moving body. We represent the time trajectories of the affine moment invariant vector as the output of a linear dynamical system driven by white noise. The problem of gait classification is then reduced to formulating distances and performing recognition in the space of linear dynamical systems. Results demonstrating the discriminate power of the proposed methods are discussed at the end.