OpenRaiser/Intern-Atlas huggingface dataset:
E2V-SDE: From Asynchronous Events to Fast and Continuous Video Reconstruction via Neural Stochastic Differential Equations
Venue: CVPR
Year: 2022

Event cameras respond to brightness changes in the scene asynchronously and independently for every pixel. Due to the properties, these cameras have distinct features: high dynamic range (HDR), high temporal resolution, and low power consumption. However, the results of event cameras should be processed into an alternative representation for computer vision tasks. Also, they are usually noisy and cause poor performance in areas with few events. In recent years, numerous researchers have attempted to reconstruct videos from events. However, they do not provide good quality videos due to a lack of temporal information from irregular and discontinuous data. To overcome these difficulties, we introduce an E2V-SDE whose dynamics are governed in a latent space by Stochastic differential equations (SDE). Therefore, E2V-SDE can rapidly reconstruct images at arbitrary time steps and make realistic predictions on unseen data. In addition, we successfully adopted a variety of image composition techniques for improving image clarity and temporal consistency. By conducting extensive experiments on simulated and real-scene datasets, we verify that our model outperforms state-of-the-art approaches under various video reconstruction settings. In terms of image quality, the LPIPS score improves by up to 12% and the reconstruction speed is 87% higher than that of ET-Net.
GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos
Venue: CVPR
Year: 2024

We address the task of generating temporally consistent and physically plausible images of actions and object state transformations. Given an input image and a text prompt describing the targeted transformation, our generated images preserve the environment and transform objects in the initial image. Our contributions are threefold. First, we leverage a large body of instructional videos and automatically mine a dataset of triplets of consecutive frames corresponding to initial object states, actions, and resulting object transformations. Second, equipped with this data, we develop and train a conditioned diffusion model dubbed GenHowTo. Third, we evaluate GenHowTo on a variety of objects and actions and show superior performance compared to existing methods. In particular, we introduce a quantitative evaluation where GenHowTo achieves 88% and 74% on seen and unseen interaction categories, respectively, outperforming prior work by a large margin.
Surface Reconstruction By Dynamic Integration Of Focus, Camera Vergence, And Stereo
Authors: A. Lynn Abbott, Narendra Ahuja
Venue: ICCV
Year: 1988

This paper concerns estimation of surface maps for real scenes having a wide field of view and a wide range of depths. Much research has emphasized stereo disparity as a source of depth information. To a lesser extent, camera focus and camera vergence have also been investigated for their utility in depth recovery. We argue that these sources of visual information have mutually complementary strengths and weaknesses, and to obtain surface maps for real scenes these processes must be integrated. Such integration requires active control of camera orientations and imaging parameters to dynamically and cooperatively interleave image acquisition with surface estimation. Accordingly, a global surface map of the visual field is synthesized by systematically scanning the scene, and combining estimates of adjacent, local surface patches, each acquired by an intermediate camera configuration and having a small depth range. We present an algorithm to perform this integration, and describe its implementation on a dynamic stereo-camera imaging system. Experimental results are presented to demonstrate the superior performance of the integrated system over that of each of its components.
Using Dynamic Programming For Minimizing The Energy Of Active Contours In The Presence Of Hard Constraints
Authors: Amir A. Amini, Saeid Tehrani, Terry E. Weymouth
Venue: ICCV
Year: 1988

Energy-Minimizing Active Contour Models (snakes) have recently been proposed by Kass et al. [8] as a top-down mechanism for locating features of interest in images. The Kass et al.’s algorithm involves four steps: setting up a variational integral on the continuous plane, deriving a pair of Euler equations, discretizing them, and solving the discrete equations iteratively until convergence. This algorithm suffers from a number of problems. We discuss these problems and present an algoIithm for active contours based on dynamic programming. The optimization problem is set up as a discrete multi-stage decision process and is solved by a “time-delayed” discrete dynamic programming algorithm. This formulation leads to a stable behavior for the active contours over iterations, in addition to allowing for hard constraints to be enforced on the behavior of the solution. Results of the application of the proposed algorithm to real images is presented.
Surface Reconstruction From Image Sequences
Authors: H. Harlyn Baker
Venue: ICCV
Year: 1988

This article presents a facility developed in support of image sequence procesiiiig for an autonoliious x-eliicle. As in man the success of the sequence analysis tlel?ends 011 t,he ability to operate explicitly in both space and tinic, and 011 exploiting the massive redundancy present iii the hundreds of view that can lie obtained when moving through a scene. The mechanism in our analysis that integrates the is a 3-D surrace-buildins process called the il'envzrzg Wall this paper focuses on the construction and use of siirfaces from this process In robotic navigation mork the surfaces represent the space-time evolulzon of scene images, and this rcpresen(ation, in coiijuiictioii with geometric constraints, allows us to determine tlie thrcc-dimensional structure of a scene. In other domains where there is a similar gradual evolution of data oyer a third dimension, in medical tomography or scale-space studies, for example, the surfaces constructed by the Weaving Wall are iiniiiediately of value for their topographic structure. The designs of both the surface-building and scene-rcconstriictiiig processes make them well-suited for real-time operation given appropriate hardware The principlc hehiiid tlie \Veaviiig TT.sl1 may have a dramatic impact 011 vision; medical iniagiiig, antl other fields.
The Feasibility Of Motion And Structure Computations
Authors: John L. Barron, Allan D. Jepson, John K. Tsotsos
Venue: ICCV
Year: 1988

We address the problem of interpreting image velocity fields generated by a moving monocular observer viewing a stationary environment under perspective projection to obtain 3-D information about the relative motion of the observer (egomotion) and the relative depth of environmental surface points (environmental layout). The algorithm presented in this paper involves computing motion and structure from a spatio-temporal distribution of image velocities that are hypothesized to belong to the same 3-D planar surface. However, the main result of this paper is not just another motion and structure algorithm that exhibits some novel features but rather an extensive error analysis of the algorithm’s preformance for various types of noise in the image velocities. Waxman and Ullman [83] have devised an algorithm for computing motion and structure using image velocity and its 1st and 2d order spatial derivatives at one image point. We generalize this result to include derivative information in time as well. Further, we show the approximate equivalence of reconstruction algorithms that use only image velocities and those that use one image velocity and its 1st and/or 2”d spatio-temporal derivatives at one image point. The main question addressed in this paper is: “How accurate do the input image velocities have to be?’ or equivalently, “How accurate does the input image velocity and its I~ and 2& order derivatives have to be?“. The answer to this question involves worst case error analysis. We end the paper by drawing some conclusions about the feasibility of motion and structure calculations in general. I.1 Introduction In this paper, we present a algorithm for computing the motion and strncture parameters that describe egomotion and environmental layout from image velocity fields generated by a moving monocular observer viewing a stationary environment. Egomotion is defined as the motion of the observer relative to his environment and can be described by 6 parameters; 3 dvth-scaled translational parameters, Z and 3 rotation parameters, o. Environmental layout refers to the 3-D shape and location of objects in the environment. For monocular image sequences, en$ronmental layout is described by the normalized surface gradient, a, at each image point. To determine these motion and structure parameters we derive nonlinear equations relating image velocity at some image int ?(?*,t ‘) to the underlying motion and structure parameters at (P,c). The computaP tion of egomotion and environmental layout from image velocity is sometimes called the reconstruction problem; we reconstruct the observer’s motion, and the layout of his environment, from (timevarying) image velocity. A lot of research has been devoted to
A Theory On Optical Velocity Fields And Ambiguous Motion Of Curves
Authors: Fredrik Bergholm
Venue: ICCV
Year: 1988

We analyze curves in motion. First, assume that we do not know the motion of points in the image plane, i.e. the optical flow. Then, define “ambiguous curves” as those that can give us nothing but ambiguous information about the underlying motion and - hence - the optical flow. The importance of a catalogue of am.bzguoas ciirues lies in telling us under what circumstances optical flow, motion and structure parameters cannot be extracted from moving curves (without point-to-point correspondence), irrespective of the method used. We give a catalogue of ambiguous curves , specializing to a planar patch in motion. Ambiguous curves (excluding the case of a single line) correspond either to a one- or a two-parameter family of solution coefficients, so the dimensionality of the non-unique solution can be predicted. Second, we wish to find analytical descriptions of the optical flow arising from objects in rigid motion, or, deforming linearly. M‘e stress the importance of using visual directions for optical flow descriptions ~ instead of image points. Modelling visual directions as elastic strings attached to a moving plane, we get a unique 3 x 3 matrix P describing the incremental transformation of visual directions. This description is more general than image point descriptions, and a convenient transform giving us concrete geometric interpretations when dealing with complicated dynamics. Using P, we can link critical-point analysis to the analysis of eigenvectors of P. The projected field lines for the optical flow, for a moving planar patch, can easily be derived from P. The two subjects, ambiguous curves and field lines of the optical flou are actually two sides of the same coin.
Robust Window Operators
Authors: Paul J. Besl, Jeffrey B. Birch, Layne T. Watson
Venue: ICCV
Year: 1988

It is a common practice in computer vision and image processing to convolve rectangular constant coefficient windows with digital images to perform local smoothing and derivative estimation for edge detection and other purposes. If all data points in each image window belong to the same statistical population, this practice is reasonable and fast. But, as is well known, constant coefficient window operators produce incorrect results if more than one statistical population is present within a window, for example, if a gray-level or gradient discontinuity is present. This paper shows one way to apply the theory of robust statistics to the data smoothing and derivative estimation problem. A robust window operator is demonstrated that preserves gray-level and gradient discontinuities in digital images as it smooths and estimates derivatives.
Synergistic Smooth Surface Stereo
Authors: Terrance E. Boult, Liang-Hua Chen
Venue: ICCV
Year: 1988

This paper presents a new algorithm for stereo matching. The algo- rithm combines what are generally three processes, feature matching, surface reconstruction, and segmentation of world surfaces, in a consis- tent and synergistic way. By integrating these phases, which are usually sequential, the algorithm can make use of the current surface approxi- mation to disambiguate potential matches. This results in higher data densities, a consistency of interpretation, and greater system flexibility. Examples of the algorithm are presented on real and synthetic images, including a scene with a transparent surface.
Admissibility Of Constraint Functions In Relaxation Labeling
Authors: H. Isil Bozma, James S. Duncan
Venue: ICCV
Year: 1988

In relaxation labeling, the domain model is incorporated into the system through the constraint functions ( compatibility coefficients). The selection of these functions is shown to be critical. In particular, for certain choices of the contraint functions, some processes will have a single nontrivial convergence point regardless of the initial labeling. Criteria on the constraint functions which result in a single convergence point are presented for those relaxation labeling processes whose formulations are within the framework of an optimization problem. We then present the necessary and sufficient conditions on the constraint €unctions under which the first of these sufficiency criteria is satisfied. The othpr conditions can be tested using standard techniques. These criteria yield a formal guideline for detecting some of the inadmi-ible constraint functions.
How the Delaunay Triangulation Can Be used For Representing Stereo Data
Authors: Elisabeth Le Bras-Mehlman, Michel Schmitt, Olivier D. Faugeras, Jean-Daniel Boissonnat
Venue: ICCV
Year: 1988

This article proposes a coherent method of interpolating 3D data obtained for example by stereo, with a polyhedral surface by means of the Delaunay Triangulation. We first give some theoretical properties concerning the approximations of sampled objects we obtain when the sampling rate increases, based on the study of their skeleton using some tools of Mathematical Morphology. Then, we present the algorithms and their complexity analysis which yield both a surface representation of objects and a volume representation of free space which may be useful in Robotics. This goal is achieved by means of a simple visibility criterion. The method is intrinsically discontinuity preserving and can be used for the integration of multiple viewpoints.
Detecting Specular Reflections Using Lambertian Constraints
Authors: Gavin Brelstaff, Andrew Blake 0001
Venue: ICCV
Year: 1988

Specularities-briglit image regions formed by specular reflectioir-are likely to be mistalien for genuine surface ma,rking.j by processes that perform photometric analysis, derive motion fields from optical flow or estimate depth from binocular stereo. In addit8iori they can be uscd to infer surface geometry. This paper describes a scheme for detecting specularities based 011 a cliaracterisatioil of Larnbertian surfaces. Real surfaces are not simple composites of Lambertiaii and specular sur- faces hut evidence suggests that away from specrilnrities they do not deviate from the Lainbertian model by more than a factor of 3 or so. This proves adequate to serve as a constraint for specularity detection. Two independent tests identify iinage regions where tlie constraint is violated. Tlie detector is shown to performs well on a varicty of real images. Applications: improving binocular stereo and inferring surface geometry are described using specularities produced by the detector. ACI(NOWLEI)GEI\IENTS: Work done at:L)ept. Conip. Sci, TJniv. Edin- burgh; Studenthip: (gjh) IDM UKSC & SERC; Grant Pics: AIVRU Sliefield; IIelp: A.Zisserinan&C.hlarinos.
Motion And Depth From Binocular Orthographic Views
Authors: Homer H. Chen
Venue: ICCV
Year: 1988

The paper describes a method using orthographic views that are taken from two cameras to determine the motion and depth of a rigid object. The cameras are widely separated in space and hence they may observe different object points. By tracking the object points for each camera, we show how the motion and depth of the object can be determined in two views when the relative position and orientation between the cameras are known. Both exact and noisy cases are discussed.
Modal Control Of An Attentive Vision System
Authors: James J. Clark, Nicola J. Ferrier
Venue: ICCV
Year: 1988

Cambridge, MA A vision system for use in a mobile robot system, or in n fixed multi-tasking industrial robot requires attentive control. Attentive control refers to the process by which the direction of gaze of the visual sensors are determined, nlong with the determination of what processing is required to be applied to the sensed images based on the goals of the robot and the tasks it is performing. This paper describes the implementation of a. mnt,ion control system which allows the attentive control of a binocular vision system. Attentive inputs to the system specify the type of visual feedback that the oculo-motor control system will use. The MDL language developed by Brockett [7] i8 used to communicate between the attentive planner and the motion controller.
On The Congruence Of Noisy Images To Line Segment Models
Authors: Ingemar J. Cox, Joseph B. Kruskal
Venue: ICCV
Year: 1988

An algorithm is described for matching two-dimensional images that does not depend on the extraction of features, a step which is often difficult or impractical. It combines computational efficiency with robustness against noisy data. It has been tested with two kinds of data. The algorithm matches a two-dimensional image, in the form of a set of points, to a line segment model. It estimates a positive congruence (i.e., a translation plus a rotation) that moves the image onto the model. The algorithm is most appropriate for matching when the required congruence is small, and it seems very well suited to motion-tracking. Experimental results have been obtained for two kinds of images: (1) points obtained from an optical range finder, and (2) edge points extracted from an intensity image. Only the former results are presented.
Temporal Edges: The Detection Of Motion And The Computation Of Optical Flow
Authors: James H. Duncan, Tsai-Chia Chou
Venue: ICCV
Year: 1988

A new method for the detection of motion and the computation of optical flow is presented. In the first step of the calculation the intensity history at each pixel is convolved with the second derivative in time of a temporal Gaussian smoothing function. The zero crossings in a single frame of the resulting function indicate the positions of moving edges. Spatial and temporal derivatives of the function at the zero-crossing locations are then used to compute the component of the flow that is normal to the zero-crossing contours. Both the detection of motion and the computation of the normal velocity are insensitive to slow temporal and spatial changes in the image intensity that are caused by illumination effects rather than motion. A framework in which to relate the present work to a number of gradient based flow measurement techniques is also presented.
A Nonlinear Approach To The Motion Correspondence Problem
Authors: Sergei Fogel
Venue: ICCV
Year: 1988

Changes in successive images from a time-varying image sequence of a scene can be characterized by velocity vector fields. In this paper the estimate of the velocity vector field is determined as a compromise in the attempt to satisfy the following two sets of constraints in addition to the regularization constraints: the optical flow constraints, which relate the values of the time-varying image function at corresponding points of the successive images of the sequence, and the directional smoothness constraints, which relate the values of the neighboring velocity vectors. To' achieve such a compromise, we introduce the system of nonlinear (equations of the unknown estimate of the velocity vector field. The optical flow and smoothness constraints are selectively suppressed in the neighborhoods of the occlusion boundaries. The last is accomplished by attaching a weight to each constraint. The spatial variations in the values of the time-varying image function near corresponding points of the successive images of the sequence, with the correspondence specified by a current estimate of tlhe velocity vector field, and variations in the current estimate of the velocity vectors themselves are implicitly used to adjust the weight function.
A Novel Approach To Colour Constancy
Authors: David A. Forsyth
Venue: ICCV
Year: 1988

By approaching colour constancy as a problem of predicting colour appearance, we derive the colour constancy equation, which we use to enumerate those properties of illuminant and surface reflectance required for colour constancy. We then use a physical realisability constraint on surface reflectances to construct the set of illuminants under which the image observed can have arisen. Two distinct algorithms arise from employing this constraint in conjunction with the colour constancy equation: the first corresponds to normalisation according to a coefficient rule, the second is considerably more complex, and allows a large number of parameters in the illuminant to be recovered. The simpler algorithm has been tested extensively on images of real Mondriaan’s, taken under different coloured lights and displays good constancy. The results also indicate that good constancy requires that receptoral gain be controlled.
Color From Black And White
Authors: Brian V. Funt, Jian Ho
Venue: ICCV
Year: 1988

Color constancy can be achieved by analyzing the chromatic aberration in an image. Chromatic aberration spatially separates light of different wavelengths and this allows the spectral power distribution of the light to be extracted. This is more information about the light than is registered by the cones of the human visual system or by a color television camera; and, using it, we show how color constancy, the separation of reflectance from illumination, can be achieved. As examples, we consider grey-level images of (a) a colored dot under unknown illumination, and (b) an edge between two differently colored regions under unknown illumination. Our first result is that in principle we can determine completely the spectral power distribution of the reflected light from the dot or, in the case of the color edge, the difference in the spectral power distributions of the light from the two regions. By employing a finite-dimensional linear model of illumination and surface reflectance, we obtain our second result, which is that the spectrum of the reflected light can be uniquely decomposed into a component due to the illuminant and another component due to the surface reflectance. This decomposition provides the complete spectral reflectance function, and hence color, of the surface as well as the spectral power distribution of the illuminant. Up to the limit of the accuracy of the finite-dimensional model, this effectively solves the color constancy problem.