nils-herrmann/clean_scirepeval huggingface dataset:
Inpainting from multiple views

Inpainting refers to the task of filling in missing or damaged regions of an image. In this paper we are interested in the inpainting problem where the missing regions are so large that local inpainting methods fail. As an alternative to the local principle, we make use of other images with related global information to enable a reasonable inpainting. Our method has roughly three phases: landmark matching, interpolation, and copying. The experimental results obtained are promising.
Structure from Motion Causally Integrated Over Time

We describe an algorithm for reconstructing three-dimensional structure and motion causally, in real time from monocular sequences of images. We prove that the algorithm is minimal and stable, in the sense that the estimation error remains bounded with probability one throughout a sequence of arbitrary length. We discuss a scheme for handling occlusions (point features appearing and disappearing) and drift in the scale factor. These issues are crucial for the algorithm to operate in real time on real scenes. We describe in detail the implementation of the algorithm, which runs on a personal computer and has been made available to the community. We report the performance of our implementation on a few representative long sequences of real and synthetic images. The algorithm, which has been tested extensively over the course of the past few years, exhibits honest performance when the scene contains at least 20-40 points with high contrast, when the relative motion is "slow" compared to the sampling frequency of the frame grabber (30 Hz), and the lens aperture is "large enough" (typically more than 30/spl deg/ of visual field).
DEFORMOTION: Deforming Motion, Shape Average and the Joint Registration and Segmentation of Images

What does it mean for a deforming object to be "moving" ? How can we separate the overall motion (a finite-dimensional group action) from the more general deformation (a diffeomorphism)? In this paper we propose a definition of motion for a deforming object and introduce a notion of "shape average" as the entity that separates the motion from the deformation. Our definition allows us to derive novel and efficient algorithms to register non-equivalent shapes using region-based methods, and to simultaneously approximate and register structures in grey-scale images. We also extend the notion of shape average to that of a "moving average" in order to track moving and deforming objects through time.
Learning Shape from Defocus

We present a novel method for inferring three-dimensional shape from a collection of defocused images. It is based on the observation that defocused images are the null-space of certain linear operators that depend on the three-dimensional shape of the scene as well as on the optics of the camera. Unlike most current work based on inverting the imaging model to recover the "deblurred" image and the shape of the scene, we approach the problem from a new angle by collecting a number of deblurred images, and estimating the operator that spans their left null space directly. This is done using a singular value decomposition. Since the operator depends on the depth of the scene, we repeat the procedure for a number of different depths. Once this is done, depth can be recovered in real time: the new image is projected onto each null-space, and the depth that results in the smallest residual is chosen. The most salient feature of this algorithm is its robustness: not only can one learn the operators with one camera and then use them to successfully retrieve depth from images taken with another camera, but one can even learn the operators from simulated images, and use them to retrieve depth from real images. Thus we train the system with synthetic patterns, and then use it on real data without knowledge of the optics of the camera. Another attractive feature is that the algorithm does not rely on a discretization or an approximation of the radiance of the scene (the "deblurred" image). In fact, the operator we recover is finite-dimensional, but it arises as the orthogonal projector of a semiinfinite operator that maps square-integrable radiance distributions onto images. Thus, the radiance is never approximated or represented via a finite set of filters. Instead, the rank of the operator learned from real data provides an estimate of the intrinsic dimensionality of the radiance distribution of real images. The algorithm is optimal in the sense of L2 and can be implemented in real time.
Stereoscopic Segmentation

We cast the problem of multiframe stereo reconstruction of a smooth shape as the global region segmentation of a collection of images of the scene. Dually, the problem of segmenting multiple calibrated images of an object becomes that of estimating the solid shape that gives rise to such images. We assume that the radiance has smooth statistics. This assumption covers Lambertian scenes with smooth or constant albedo as well as fine homogeneous textures, which are known challenges to stereo algorithms based on local correspondence. We pose the segmentation problem within a variational framework, and use fast level set methods to approximate the optimal solution numerically. Our algorithm does nor work in the presence of strong textures, where traditional reconstruction algorithms do. It enjoys significant robustness to noise under the assumptions it is designed for.
Dynamic textures

Dynamic textures are sequences of images of moving scenes that exhibit certain stationarity properties in time; these include sea-waves, smoke, foliage, whirlwind but also talking faces, traffic scenes etc. We present a novel characterization of dynamic textures that poses the problems of modelling, learning, recognizing and synthesizing dynamic textures on a firm analytical footing. We borrow tools from system identification to capture the "essence" of dynamic textures; we do so by learning (i.e. identifying) models that are optimal in the sense of maximum likelihood or minimum prediction error variance. For the special case of second-order stationary processes we identify the model in closed form. Once learned, a model has predictive power and can be used for extrapolating synthetic sequences to infinite length with negligible computational cost. We present experimental evidence that, within our framework, even low dimensional models can capture very complex visual phenomena.
Dynamic Data Factorization

We present a technique for reducing the dimensionality of a s tationary time series with an arbitrary covariance sequenc e. This problem is solved in closed form in a way that is asymptot ically efficient (i.e. optimal in the sense of maximum likeli hood). Our technique can be seen as a special case of a more general th ory of subspace identification borrowed from the field of dynamical systems. Although our scheme only captures secon d-order statistics, we suggest extensions to model higherorder moments. Our technique can be used to perform factorization under the constraint that one of the factors evolves accordi ng to a dynamical model. Applications range from structure fro m motion to the analysis of dynamic textures.
Controlling Beauty - Perceptual Representation and Control of Geometric Shape Spaces

We consider the problem of representing geometric entities in a perceptual coordinate system. A geometric object (for instance a closed contour or a surface patch) described by a finite number of parameters is mapped onto an arbitrary collection of perceptual coordinates via a psychophysical experiment, resulting in a collection of marginal probability distributions. Vice versa, a set of perceptual coordinates can be mapped onto a probability distribution of parameters, and a pointwise inverse can be constructed by tracing the mode(s) of the density. We use such a map to modify the shape of an object by controlling perceptual characteristics, as opposed to manipulating the geometry directly. We choose to test our hypothesis on a simple one-dimensional experiment to map the shape of eyeglass frames to the perceived character of the person wearing them. We believe that perception-driven geometric manipulation presents vast opportunities in a number of commercial applications of mass-customized design.
Stereoscopic segmentation

We cast the problem of multiframe stereo reconstruction of a smooth shape as the global region segmentation of a collection of images of the scene. Dually, the problem of segmenting multiple calibrated images of an object becomes that of estimating the solid shape that gives rise to such images. We assume that the radiance has smooth statistics. This assumption covers Lambertian scenes with smooth or constant albedo as well as fine homogeneous textures, which are known challenges to stereo algorithms based on local correspondence. We pose the segmentation problem within a variational framework, and use fast level set methods to approximate the optimal solution numerically. Our algorithm does nor work in the presence of strong textures, where traditional reconstruction algorithms do. It enjoys significant robustness to noise under the assumptions it is designed for.
Real-time virtual object insertion

We present a system to insert virtual objects into real image sequences in real time. The system consists of offthe- shelf hardware (a camera connected to a Pentium PC) and software to (a) automatically select and track region features despite changes in illumination, (b) estimate threedimensional position and orientation of surface patches relative to an inertial reference frame despite individual pointfeatures appearing and disappearing, (c) insert a texturemapped virtual object into the scene so as to make it appear to be part of the scene and moving with it. This is all done in real time. The multi-thread C++ code, which is readily interfaced with a frame grabber as well as Matlab for development, will be made available to the public at the demonstration.
A semi-direct approach to structure from motion

Reconstructing three-dimensional structure and motion is often decomposed into two steps: point feature correspondence and three-dimensional reconstruction. This separation often causes gross errors since correspondence relies on the brightness constancy constraint that is local in space and time. Therefore, we advocate the necessity to integrate visual information not only in time (i.e. across different views), but also in space, by matching regions-rather than points-using explicit photometric deformation models. We present an algorithm that integrates 2D region tracking and 3D motion estimation into a closed loop based on an explicit geometric and photometric model, while detecting and rejecting outlier regions that do not fit the model. Our algorithm is recursive and suitable for real-time implementation. Our experiments show that it far exceeds the accuracy and robustness of point feature-based SFM algorithms.
Recognition of human gaits

We pose the problem of recognizing different types of human gait in the space of dynamical systems where each gait is represented Established techniques are employed to track a kinematic model of a human body in motion, and the trajectories of the parameters are used to learn a representation of a dynamical system, which defines a gait. Various types of distance between models are then computed These computations are non trivial due to the fact that, even for the case of linear systems, the space of canonical realizations is not linear.
Stereoscopic shading: integrating multi-frame shape cues in a variational framework

We address the problem of integrating multi-frame stereo and shading cues within the framework of optimization in the infinite-dimensional space of piecewise smooth surfaces. Cue integration then reduces to the determination of regions where prior assumptions on the reflectance of the surfaces can be enforced. By combining cues, our formulation allows defining a well-posed problem even when reconstruction from stereo or shading in isolation would be ill-posed. For a simplified model we prove the necessary conditions for optimality, and propose an iterative optimization algorithm, which we implement using ultra-narrowband level set methods.
Shape and Radiance Estimation from the Information-Divergence of Blurred Images

We formulate the problem of reconstructing the shape and radiance of a scene as the minimization of the information divergence between blurred images, and propose an algorithm that is provably convergent and guarantees that the solution is admissible, in the sense of corresponding to a positive radiance and imaging kernel. The motivation for the use of information divergence comes from the work of Csiszar , while the fundamental elements of the proof of convergence come from work by Snyder et al. , extended to handle unknown imaging kernels (i.e. the shape of the scene).
Real-time 3D motion and structure of point features: a front-end system for vision-based control and interaction

We present a system that consists of one camera connected to a personal computer that can (a) select and track a number of high-contrast point features on a sequence of images, (b) estimate their three-dimensional motion and position relative to an inertial reference frame, assuming rigidity, (c) handle occlusions that cause point-features to disappear as well as new features to appear. The system can also (d) perform partial self-calibration and (e) check for consistency of the rigidity assumption, although these features are not implemented in the current release. All of this is done automatically and in real-time (30 Hz) for 40-50 point features using commercial off-the-shelf hardware. The system is based on an algorithm presented by Chiuso et al. (2000), the properties of which have been analyzed by Chiuso and Soatto (2000). In particular, the algorithm is provably observable, provably minimal and provably stable- under suitable conditions. The core of the system, consisting of C++ code ready to interface with a frame grabber as well as Matlab code for development, is available at http://ee.wustl.edu/-soatto/research.html. We demonstrate the system by showing its use as (1) an ego-motion estimator, (2) an object tracker, and (3) an interactive input device, all without any modification of the system settings.
3-D Motion and Structure from 2-D Motion Causally Integrated over Time: Implementation

The causal estimation of three-dimensional motion from a sequence of two-dimensional images can be posed as a nonlinear filtering problem. We describe the implementation of an algorithm whose uniform observability, minimal realization and stability have been proven analytically in . We discuss a scheme for handling occlusions, drift in the scale factor and tuning of the filter. We also present an extension to partially calibrated camera models and prove its observability. We report the performance of our implementation on a few long sequences of real images. More importantly, however, we have made our real-time implementation - which runs on a personal computer - available to the public for first-hand testing.
Monte Carlo filtering on Lie groups

We propose a nonlinear filter for estimating the trajectory of a random walk on a matrix Lie group with constant computational complexity. It is based on a finite-dimensional approximation of the conditional distribution of the state-given past measurements-via a set of fair samples, which are updated at each step and proven to be consistent with the updated conditional distribution. The algorithm proposed, like other Monte Carlo methods, can in principle track arbitrary distributions evolving on arbitrarily large state spaces. However, several issues concerning sample impoverishment need to be taken into account when designing practical working systems.
A geometric approach to blind deconvolution with application to shape from defocus

We propose a solution to the generic "bilinear calibration-estimation problem" when using a quadratic cost function and restricting to (locally) translation-invariant imaging models. We apply the solution to the problem of reconstructing the three-dimensional shape and radiance of a scene from a number of defocused images. Since the imaging process maps the continuum of three-dimensional space onto the discrete pixel grid, rather than discretizing the continuum we exploit the structure of maps between (finite-and infinite-dimensional) Hilbert spaces and arrive at a principled algorithm that does not involve any choice of basis or discretization. Rather, these are uniquely determined by the data, and exploited in a functional singular value decomposition in order to obtain a regularized solution.
Filtering on Lie Groups

We propose a nonlinear lter for estimating the trajectory of a random walk on a matrix Lie group with constant computational complexity. It is based on a nite-dimensional approximation of the conditional distribution of the state given past measurements via a set of fair samples, which are updated at each step and proven to be consistent with the updated conditional distribution. The algorithm proposed, like other Monte Carlo methods, can in principle track arbitrary distributions evolving on arbitrarily large state spaces. However, several issues concerning sample impoverishment need to be taken into account when designing practical working systems.
The Accommodation Cue in Vision

Any imaging system, such as the human eye, a video-camera, a CT scanner or a telescope, entails a map of the three-dimensional environment onto the two-dimensional surface of an imaging sensor. Images of any such map are characterized by a loss of information along one spatial dimension. The visual system of primates has evolved to exploit two-dimensional images to retrieve a representation of the environment that can be used to perform spatial control tasks crucial to survival.
On observing shape from defocused images

Accommodation cues are measurable properties of an image that are associated with a change of the geometry of the imaging device. To what extent can three-dimensional shape be reconstructed using accommodation cues alone? This question is fundamental to the problem of reconstructing "shape from focus" (SFF) and "shape from defocus" (SFD) for applications in microscopy, image restoration (de-blurring) and visualization. We address it by studying the observability of accommodation cues in an analytical framework that is independent of the imaging device.
Euclidean reconstruction and reprojection up to subgroups

The necessary and sufficient conditions for being able to estimate scene structure, motion and camera calibration from a sequence of images are very rarely satisfied in practice. What exactly can be estimated in sequences of practical importance, when such conditions are not satisfied? In this paper we give a complete answer to this question. For every camera motion that fails to meet the conditions, we give explicit formulas for the ambiguities in the reconstructed scene, motion and calibration. Such a characterization is crucial both for designing robust estimation algorithms (that do not try to recover parameters that cannot be recovered), and for generating novel views of the scene by controlling the vantage point. To this end, we characterize explicitly all the vantage points that give rise to a valid Euclidean reprojection regardless of the ambiguity in the reconstruction. We also characterize vantage points that generate views that are altogether invariant to the ambiguity. All the results are presented using simple notation that involves no tensors nor complex projective geometry, and should be accessible with basic background in linear algebra.
Optimal structure from motion: local ambiguities and global estimates

We present an analysis of SFM from the point of view of noise. This analysis results in an algorithm that is provably convergent and provably optimal with respect to a chosen norm. In particular, we cast SFM as a nonlinear optimization problem and define a bilinear projection iteration that converges to fixed points of a certain cost-function. We then show that such fixed points are "fundamental", i.e. intrinsic to the problem of SFM and not an artifact introduced by our algorithm. We classify and characterize geometrically local extrema, and we argue that they correspond to phenomena observed in visual psychophysics. Finally, we show under what conditions it is possible-given convergence to a local extremum-to "jump" to the valley containing the optimum; this leads us to suggest a representation of the scene which is invariant with respect to such local extrema.
Observability of shape from focus

We consider a closed-loop system consisting of a commercial video camera with auto-focus. We view the position of the lens as the controlled input, the brightness array on the CCD sensor as the measured output, and the auto-focus' task to maintain a region of the image "in focus". As we see, the (non-linear and distributed) input/output map that describes the imaging process depends upon the shape of the scene being viewed. The aim of the paper is to study the observability of three-dimensional shape from two-dimensional images of a static scene obtained by controlling the focus of the camera.
Optimal and Suboptimal Structure From MotionStefano

\Structure From Motion" (SFM) refers to the problem of estimating three-dimensional information about the environment from the motion of its two-dimensional projection onto a surface (for instance the retina). Noise plays an important role in this problem, but it has been addressed only marginally in more than twenty years of research. We present an analysis of SFM from the point of view of noise. This analysis results in algorithms that are provably convergent and provably optimal with respect to a chosen norm. In particular, we cast SFM as a nonlinear optimization problem and deene a bilinear projection iteration that converges to xed points of a certain cost-function. We then show that such xed points are \fundamental", i.e. are intrinsic to the problem of SFM and not an artifact introduced by our algorithms. We classify and interpret geometrically local extrema, and we argue that they correspond to phenomena observed in visual psychophysics. Finally, we show under what conditions it it is possible, given convergence to a local extremum, to \jump" to the valley containing the optimum.
Visual path following by recursive spline updating

In this paper we address the problem of tracking an unknown contour by commanding the translational and rotational velocities of a nonholonomic vehicle moving on a plane. A TV camera provides measurements of the distance of the contour from the x-axis of the moving frame (fixed with the vehicle). We discuss an online path planning control strategy which leads the vehicle to land softly at a desired point ahead on the unknown curve and to stay on the curve once it has been reached. This strategy could be seen as an explicit model-based predictive control scheme.
A lagrangian formulation of nonholonomic path following

We address the problem of following an unknown planar contour with a nonholonomic vehicle based on visual feedback. The control task is to keep a point of the vehicle as close as possible to the contour for a choice of norm. A camera mounted on-board the vehicle provides measurements of the contour. We formulate the problem and compute the control law in a moving reference frame modeling the evolution of the contour as seen by an observer sitting on the vehicle. The result is an on-line path planning strategy and a predictive control law which leads the vehicle to land softly on the unknown curve. Depending on the choice of the tracking criterion, the controller can exhibit non-trivial behaviors including automatic maneuvering.