nils-herrmann/clean_scirepeval huggingface dataset:
Mumford-Shah for Segmentation and Stereo

We begin by presenting an active contour model which utilizes the Mumford-Shah energy functional for the purpose of piecewise smooth image segmentation. We then show how the use of simultaneous piecewise smooth image segmentation on a set of calibrated 2D images of a common 3D scene may be utilized for reconstructing the unknown shapes and radiances of scene objects. To do so, we must lift the the support of the unknown smooth functions in the traditional Mumford-Shah framework, which will now represent the unknown radiance of scene objects in this application, onto a manifold which will represent the unknown shape of scene objects. This constitutes a significant mathematical departure from the traditional Mumford-Shah model since the unknown functions now live directly on the unknown geometric surfaces rather than their surrounding ambient space. The intution, however, follows from the original model in that the estimates must closely match the observed data while maintaining a high degree of smoothness both in the estimated functions as well as the estimated geometry.
Tales of shape and radiance in multiview stereo

To what extent can three-dimensional shape and radiance be inferred from a collection of images? Can the two be estimated separately while retaining optimality? How should the optimality criterion be computed? When is it necessary to employ an explicit model of the reflectance properties of a scene? In this paper we introduce a separation principle for shape and radiance estimation that applies to Lambertian scenes and holds for any choice of norm. When the scene is not Lambertian, however, shape cannot be decoupled from radiance, and therefore matching image-to-image is not possible directly. We employ a rank constraint on the radiance tensor, which is commonly used in computer graphics, and construct a novel cost functional whose minimization leads to an estimate of both shape and radiance for nonLambertian objects, which we validate experimentally.
A Pseudo-distance for Shape Priors in Level Set Segmentation

We study the question of integrating prior shape knowledge into level set based segmentation methods. In particular, we investigate dissimilarity measures for shapes encoded by the signed distance function. We consider extensions and improvements of existing measures. As a result, we propose a novel dissimilarity measure which constitutes a pseudo-distance. Compared to alternative approaches, this measure is symmetric and not biased toward small area. Based on this pseudo-distance, we propose a shape prior for level set segmentation methods which is pose invariant. In numerical experiments, we demonstrate that the resulting prior permits the segmentation of corrupted versions of a familiar object which is independent of the pose of the object. Moreover, we demonstrate the advantage of the symmetric formulation of the dissimilarity measure when segmenting corrupted images of known objects which consist of multiple components.
An Invitation to 3-D Vision: From Images to Geometric Models

This book introduces the geometry of 3-D vision, that is, the reconstruction of 3-D models of objects from a collection of 2-D images. It details the classic theory of two view geometry and shows that a more proper tool for studying the geometry of multiple views is the so-called rank consideration of the multiple view matrix. It also develops practical reconstruction algorithms and discusses possible extensions of the theory.
Editable dynamic textures

We present a simple and efficient algorithm for modifying the temporal behavior of "dynamic textures," i.e. sequences of images that exhibit some form of temporal regularity, such as flowing water, steam, smoke, flames, foliage of trees in wind. The main goal is to design algorithms for synthesizing and editing realistic sequences of images of dynamic scenes that exhibit some form of temporal stationarity. This is an image-based rendering task, and in particular we are interested in synthesizing the temporal behavior of the scene.
Variational space-time motion segmentation

We propose a variational method for segmenting image sequences into spatiotemporal domains of homogeneous motion. To this end, we formulate the problem of motion estimation in the framework of Bayesian inference, using a prior which favors domain boundaries of minimal surface area. We derive a cost functional which depends on a surface in space-time separating a set of motion regions, as well as a set of vectors modeling the motion in each region. We propose a multiphase level set formulation of this functional, in which the surface and the motion regions are represented implicitly by a vector-valued level set function. Joint minimization of the proposed functional results in an eigenvalue problem for the motion model of each region and in a gradient descent evolution for the separating interface. Numerical results on real-world sequences demonstrate that minimization of a single cost functional generates a segmentation of space-time into multiple motion regions.
Seeing beyond occlusions (and other marvels of a finite lens aperture)

We present a novel algorithm to reconstruct the geometry and photometry of a scene with occlusions from a collection of defocused images. The presence of a finite lens aperture allows us to recover portions of the scene that would be occluded in a pin-hole projection, thus "uncovering" the occlusion. We estimate the shape of each object (a surface, including the occluding boundaries), and its radiance (a positive function defined on the surface, including portions that are occluded by other objects).
Estimation of 3D Surface Shape and Smooth Radiance from 2D Images: A Level Set Approach

We cast the problem of shape reconstruction of a scene as the global region segmentation of a collection of calibrated images. We assume that the scene is composed of a number of smooth surfaces and a background, both of which support smooth Lambertian radiance functions. We formulate the problem in a variational framework, where the solution (both the shape and radiance of the scene) is a minimizer of a global cost functional which combines a geometric prior on shape, a smoothness prior on radiance and a data fitness score. We estimate the shape and radiance via an alternating minimization: The radiance is computed as the solutions of partial differential equations defined on the surface and the background. The shape is estimated using a gradient descent flow, which is implemented using the level set method. Our algorithm works for scenes with smooth radiances as well as fine homogeneous textures, which are known challenges to traditional stereo algorithms based on local correspondence.
Structure from motion for scenes without features

We describe an algorithm for reconstructing the 3D (three-dimensional) shape of the scene and the relative pose of a number of cameras from a collection of images under the assumption that the scene does not contain photometrically distinct "features". We work under the explicit assumption that the scene is made of a number of smooth surfaces that radiate constant energy isotropically in all directions, and setup a region-based cost functional that we minimize using local gradient flow techniques.
Variational methods for shape reconstruction in computer vision

This dissertation addresses the problem of inferring the three-dimensional shape and radiance of a scene from a collection of calibrated images taken from different viewpoints. The problem is known as “multi-view stereo” and has been studied extensively in the Computer Vision literature. Most existing multi-view stereo algorithms rely on identification of corresponding points or regions among images. They work effectively only when scenes are Lambertian and have textured radiances, because lack of Lambertianity or absence of texture in the radiance leads to an ill-posed correspondence problem. Targeting scenes with more complex photometric properties, we propose a framework of comparing the images with a model of the scene, which avoids the ill-posed correspondence problem. The model is comprised of a component for shape and a component for radiance. We model the shape as a collection of smooth surfaces and model the radiance based on the underlying reflectance properties. Within this framework, we present two stereo reconstruction algorithms, one for non-Lambertian scenes and the other for Lambertian scenes with smooth radiances. 
In addressing non-Lambertian scenes, we propose a novel model for the radiance. At the core of the model lies a rank constraint of the radiance tensor field and a discrepancy function that measures how well the image data satisfy the constraint. We exploit this model to reconstruct both the shape and radiance. The latter can be used to generate novel images that preserve the non-Lambertian appearance of the scene. 
In addressing Lambertian scenes with smooth radiances, we propose to model the radiance as a smooth function defined on an unknown surface. We then estimate both the shape and radiance by optimizing a cost functional which combines a geometric prior on shape, a smoothness prior on radiance and a data fitness score. Our algorithm jointly segments images in a region-based fashion. This scheme does not require computing image gradients and therefore is robust to image noise.
Dynamic texture segmentation

We address the problem of segmenting a sequence of images of natural scenes into disjoint regions that are characterized by constant spatio-temporal statistics. We model the spatio-temporal dynamics in each region by Gauss-Markov models, and infer the model parameters as well as the boundary of the regions in a variational optimization framework. Numerical results demonstrate that - in contrast to purely texture-based segmentation schemes - our method is effective in segmenting regions that differ in their dynamics even when spatial statistics are identical.
On exploiting occlusions in multiple-view geometry

Occlusions are commonplace in man-made and natural environments; they often result in photometric features where a line terminates at an occluding boundary, resembling a "T". We show that the 2-D motion of such T-junctions in multiple views carries nontrivial information on the 3-D structure of the scene and its motion relative to the camera. We show how the constraint among multiple views of T-junctions can be used to reliably detect them and differentiate them from ordinary point features. Finally, we propose an integrated algorithm to recursively and causally estimate structure and motion in the presence of T-junctions along with other point-features.
Towards Plenoptic Dynamic Textures

We present a technique to infer a model of the spatio-temporal statistics of a collection of images of dynamic scenes seen from a moving camera. We use a time-variant linear dynamical system to jointly model the statistics of the video signal and the moving vantage point. We propose three approaches to inference, the first based on the plenoptic function, the second based on interpolating linear dynamical models, the third based on approximating the scene as piecewise planar. For the last two approaches, we also illustrate the potential of the proposed techniques with a number of experiments. The resulting algorithms could be useful for video editing where the motion of the vantage point can be controlled interactively, as well as to perform stabilized synthetic generation of video sequences.
Vision Based Navigation

We present a method for inferring the location of a robot relative to a three-dimensional map of its environment. The map, created off-line, consists of image patches, their locations in space, and their associated normal vectors. Observational data consist of one or many images taken from the robot’iCs current viewpoint (a point and orientation in space). We develop a framework for matching images of a scene (observations) to a map and show how this can be applied to the task of robot localization. Localization is posed as an optimization problem, where the observed data and the map are aligned to produce an estimate of the current pose of the robot. We provide a formalization of our model and demonstrate experimental results in unstructured environments.
3D shape from anisotropic diffusion

We cast the problem of inferring the 3D shape of a scene from a collection of defocused images in the framework of anisotropic diffusion. We propose an algorithm that can estimate the shape of a scene by inferring the diffusion coefficient of a heat equation. The method is optimal, as we pose it as the minimization of a certain cost functional based on the input images, and fast. Furthermore, we also extend our algorithm to the case of multiple images, and derive a 3D scene segmentation algorithm that can work in the presence of pictorial camouflage.
Multi-view stereo beyond Lambert

We consider the problem of estimating the shape and radiance of an object from a calibrated set of views under the assumption that the reflectance of the object is non-Lambertian. Unlike traditional stereo, we do not solve the correspondence problem by comparing image-to-image. Instead, we exploit a rank constraint on the radiance tensor field of the surface in space, and use it to define a discrepancy measure between each image and the underlying model. Our approach automatically returns an estimate of the radiance of the scene, along with its shape, represented by a dense surface. The former can be used to generate novel views that capture the non-Lambertian appearance of the scene.
Observability of Linear Hybrid Systems

We analyze the observability of the continuous and discrete states of continuous-time linear hybrid systems. For the class of jump-linear systems, we derive necessary and sufficient conditions that the structural parameters of the model must satisfy in order for filtering and smoothing algorithms to operate correctly. Our conditions are simple rank tests that exploit the geometry of the observability subspaces. For linear hybrid systems, we derive weaker rank conditions that are sufficient to guarantee the uniqueness of the reconstruction of the state trajectory, even when the individual linear systems are unobservable.
Region Matching and Tracking under Deformations or Occlusions

We present a variational approach to the problem of registering nonequivalent shapes with missing parts. Registration is achieved through the evolution of partial differential equations that simultaneously estimates the shape of the missing region, the underlying “average complete shape” and the collection of group elements (Euclidean or affine) performing the registration. Our techniques apply both to shapes, for instance represented as characteristic functions (binary images), and to grayscale images, where all intensity levels evolve simultaneously in a system of partial differential equations.
Controllable Visual Cues

This project concerns the development of analytical and computational tools for the analysis of dynamic visual processes. In particular, emphasis has been given to the estimation of visual motion for the purpose of control, the estimation of shape from moving images, the estimation of spatio-temporal image statistics for visual modeling, synthesis and recognition. The project is in its final year, and the progress has been documented extensively in the following publications [33, 36, 8, 29, 41, 32, 7, 25, 9, 40, 28, 20, 48, 22, 37, 1, 15, 2, 42, 47, 13, 45, 16, 24, 27, 30, 52, 44, 19, 12, 10, 18, 3, 39, 11, 4, 35, 17, 33, 5, 6, 14, 23, 21, 31, 49, 50, 46, 26, 38, 51, 53, 43, 34]. These include a textbook [34], a Marr Prize-winning manuscript  and a Siemens Prize (best paper award at CVPR) manuscript , and the first real-time demonstration of a system to estimate 3D structure and motion in real time on commercial off-the-shelf hardware [20]. Many of the results obtained during the course of this project have been incorporated in a new first year graduate course developed at UCLA (CS268, Machine Perception), as well as in a three-day short course outreach to industry and government titled “3D reconstruction from Video” administered through the UCLA extension in the summer of 2002 and 2003.
Vision for Control: Optimality and Usability

This final report for the grant DAADl9-99-l-0139 and the associated instrumentation grant DURlP 4425l0-2l338 describes research conducted both at Washington University in St. Louis and at the University of California, Los Angeles, from 1999 to 2003. Research has been documented in 34 publications, among which two best-paper awards (the Marr Prize, the highest recognition in the field of Computer Vision, and the Siemens Prize with the Outstanding Paper Award by the IEEE Computer Society) and a book, expected to appear in June 2003. Technology firsts include the first ever system (and still the only one) for estimating three-dimensional structure and motion of an arbitrary (static) scene in real time, and the first optimal algorithms for estimating shape from accommodation. Given the scale of the project, consisting of 70K
An algebraic geometric approach to the identification of a class of linear hybrid systems

We propose an algebraic geometric solution to the identification of a class of linear hybrid systems. We show that the identification of the model parameters can be decoupled from the inference of the hybrid state and the switching mechanism generating the transitions, hence we do not constraint the switches to be separated by a minimum dwell time. The decoupling is obtained from the so-called hybrid decoupling constraint, which establishes a connection between linear hybrid system identification, polynomial factorization and hyperplane clustering. In essence, we represent the number of discrete states n as the degree of a homogeneous polynomial p and the model parameters as factors of p. We then show that one can estimate n from a rank constraint on the data, the coefficients of p from a linear system, and the model parameters from the derivatives of p. The solution is closed form if and only if n/spl les/4. Once the model parameters have been identified, the estimation of the hybrid state becomes a simpler problem. Although our algorithm is designed for noiseless data, we also present simulation results with noisy data.
A semi-direct approach to structure from motion

The problem of structure from motion is often decomposed into two steps: feature correspondence and three-dimensional reconstruction. This separation often causes gross errors when establishing correspondence fails. Therefore, we advocate the necessity to integrate visual information not only in time (i.e. across different views), but also in space, by matching regions – rather than points – using explicit photometric deformation models. We present an algorithm that integrates image-feature tracking and three-dimensional motion estimation into a closed loop, while detecting and rejecting outlier regions that do not fit the model. Due to occlusions and the causal nature of our algorithm, a drift in the estimates accumulates over time. We describe a method to perform global registration of local estimates of motion and structure by matching the appearance of feature regions stored over long time periods. We use image intensities to construct a score function that takes into account changes in brightness and contrast. Our algorithm is recursive and suitable for real-time implementation.
Towards unsupervised detection of actions in clutter

In this paper, we describe a characterization of visual action that encodes a local photometry via a choice of interest operators and global dynamics via a realization of a stochastic dynamical model. In order to allow action detection in clutter, it is necessary for the corresponding models to have a compositional property, in that a simple action (e.g. foreground action) can be detected within a more complex one (e.g. foreground and background actions). We show that this is the case for the model we propose, which can therefore be used as a basis for building models of dynamic scenes from images without explicit supervision, by composing a complex action from a collection of elementary ones.
Region Matching with Missing Parts

We present a variational approach to the problem of registering planar shapes despite missing parts. Registration is achieved through the evolution of a partial differential equation that simultaneously estimates the shape of the missing region, the underlying "complete shape" and the collection of group elements (Euclidean or affine) corresponding to the registration. Our technique applies both to shapes, for instance represented as characteristic functions (binary images), and to grayscale images, where all intensity levels evolve simultaneously in a partial differential equation. It can therefore be used to perform "region inpainting" and to register collections of images despite occlusions. The novelty of the approach lies on the fact that, rather than estimating the missing region in each image independently, we pose the problem as a joint registration with respect to an underlying "complete shape" from which the complete version of the original data is obtained via a group action.
Two-View Segmentation of Dynamic Scenes from the Multibody Fundamental Matrix

We present a geometric approach for the analysis of dynamic scenes containing multiple rigidly moving objects seen in two perspective views. Our approach exploits the algebraic and geometric properties of the so-called multibody epipolar constraint and its associated multibody fundamental matrix, which are natural generalizations of the epipolar constraint and of the fundamental matrix to multiple moving objects. We derive a rank constraint on the image points from which one can estimate the number of independent motions and linearly solve for the multibody fundamental matrix. We prove that the epipoles of each independent motion lie exactly in the intersection of the left null space of the multibody fundamental matrix with the so-called Veronese surface. We then show that individual epipoles and epipolar lines can be uniformly and efficiently computed by using a novel polynomial factorization technique. Given the epipoles and epipolar lines, the estimation of individual fundamental matrices becomes a linear problem. Then, motion and feature point segmentation is automatically obtained from either the epipoles and epipolar lines or the individual fundamental matrices. We test the proposed approach by segmenting a real sequence.
Observability and identifiability of jump linear systems

We analyze the observability of the continuous and discrete states of a class of linear hybrid systems. We derive rank conditions that the structural parameters of the model must satisfy in order for filtering and smoothing algorithms to operate correctly. We also study the identifiability of the model parameters by characterizing the set of models that produce the same output measurements. Finally, when the data are generated by a model in the class, we give conditions under which the true model can be identified.
A model (In)validation approach to gait recognition

This paper addresses the problem of human gait recognition by applying model (in)validation techniques. The main idea is to associate to each class of gaits a nominal model and a class of bounded energy inputs. In this context, the problem of recognizing a sequence can be formulated as the problem of determining whether or not it could have been generated by a given model and its associated class of inputs. By exploiting interpolation theory results this problem can be recast into a Linear Matrix Inequality (LMI) optimization form and efficiently solved.