Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects

1 Introduction . Motion blur is a common cause of degraded image quality . It can originate from camera motion , fast object motion , long exposure times due to low light settings , or a combination of these effects . In this paper , we focus on deblurring of fast moving objects ( FMOs ) , which move over a distance larger than their size for the duration of the camera exposure of a single image . Since FMOs appear as a mixture of motion-blurred object texture and the background , our first goal is to decompose the combined appearance of the background and the one of the FMO in form of a matting mask [ 1 ] . Our main goal is to recover the shape , motion , and sharp texture of the FMO in order to explain its appearance in the input image in the best possible way ( Fig . 1 ) . The accurate reconstruction and motion estimation of FMOs enables to generate videos with temporal super-resolution , as well as FMO motion analysis and prediction . This is useful for a variety of applications like sports analysis ( e.g . soccer , tennis , baseball ) , meteorites detection in astrophysics , detecting fast moving obstacles in front of a vehicle for driver alert or autonomous driving , and general video quality enhancement or compression . Our work could eventually make FMO tracking accessible to everyone using regular cameras rather than expensive high-speed cameras , or push the capabilities of current high-speed cameras to a new level , e.g . by capturing gun shots . The majority of deblurring methods aim for the generic task of removing any kind of blur in images [ 2 , 3 , 4 ] or videos [ 5 , 6 , 7 , 8 , 9 ] . However , they perform poorly on deblurring of FMOs as 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . shown in comparisons to methods specialized for them [ 10 , 11 ] . To the best of our knowledge , all generic and all FMO-specialized methods try to tackle the deblurring task only in the 2D image domain . This limits their ability to describe object rotations around any axis that does not correspond to the camera viewing direction , or any object translation along the viewing direction . Our key idea is to model the object shape , motion , and texture in the 3D domain to better explain the input image as the projection and temporal aggregation of a continuously moving object . We recover motion as the initial 6-DoF pose of the object and the 3D translation and 3D rotation that it undergoes during the exposure time of the input image . Moreover , the proposed method is the first to estimate the 3D shape of motion-blurred objects . Although we are solving a more general and more challenging problem than previous methods , we achieve substantially better deblurring results due to the rigorous joint modeling of object shape , motion , and texture in 3D . In summary , we make the following contributions : ( 1 ) We propose a novel fast moving object deblurring method that for the first time jointly estimates from a single input image the 3D shape , texture , and motion of an object ( initial 6-DoF pose , 3D translation and 3D rotation ) . ( 2 ) We describe an effective pipeline that uses pure test-time optimization for the recovery of the object 3D shape , motion , and sharp appearance . To constrain the optimization , we propose several suitable regularizers . 2 Related work . Generic deblurring . Single-image deblurring is studied in [ 12 , 2 ] , and video deblurring in [ 9 , 5 , 6 , 7 , 8 ] . Some methods were specifically proposed for motion deblurring [ 13 , 14 , 15 ] , but they generate only a single output image or a motion flow . A more difficult task of estimating the temporal super-resolution from a single image has been addressed in [ 3 , 4 , 16 ] , but only for small motions . Below we review methods specialized to deblurring fast moving objects ( FMOs ) . Constant 2D object appearance . The image formation with motion blur of a FMO [ 10 , 17 ] was introduced as I = H ∗ F + ( 1−H ∗M ) ·B , ( 1 ) where the input image I is formed as a composition of two terms . The first term represents the blurred object as a convolution of the sharp object appearance F and the blur kernel H . The second term encodes the influence of the background B , which depends on the object matting mask M . This model assumes that the object appearance is constant throughout the whole exposure time . Tracking by Deblatting ( TbD ) [ 18 , 19 ] , from deblurring and matting , was proposed to estimate the unknown sharp object model ( F , M ) and its motion represented by H . To achieve this , Kotera et al . [ 18 ] applied ADMM variable splitting with appearance total variation and blur kernel sparsity priors . Follow-up methods improve the precision of the 2D trajectory estimation [ 20 , 21 ] . The special case of planar object with planar rotation is studied in [ 22 ] . Piece-wise constant 2D object appearance . The constant appearance model ( 1 ) was extended to piece-wise constant [ 23 ] to deal with small amounts of appearance changes over time . The appearance might change over time due to object translation in 3D , object rotation in 3D , view-specific changes due to illumination , specularities , shadows , and even object deformation , e.g . when the object bounces off the ground . The piece-wise constant model is defined as I = ∑ τ Hτ ∗ Fτ + ( 1− ∑ τ Hτ ∗Mτ ) ·B , ( 2 ) where the exposure time is split into several uniform time intervals τ ( sub-frames ) . Each sub-frame has its own appearance model ( Fτ , Mτ ) and blur kernel Hτ , which are constant within the sub-frame . TbD-3D [ 23 ] proposed to estimate these piece-wise appearances by energy minimization with several priors , such as rotational symmetry , appearance total variation , and penalizing appearance from changing rapidly across neighboring sub-frames . The blur kernel is still estimated by deblatting [ 18 ] assuming the simple constant appearance model ( 1 ) , and the sub-frame appearances are estimated in a second pass by splitting the trajectory into the desired number of pieces . Thus , TbD-3D performs well only with very small appearance changes and mostly spherical objects due to the rotational symmetry prior . For perfectly spherical objects , TbD-3D can also estimate 3D rotation by minimizing the sphere reprojection error . Then , the object-camera distance is approximated by the radius of estimated 2D masks . However , this method can not generalize to other shapes . 2D projections of any object . DeFMO [ 11 ] was the first learning-based method for FMO deblurring . They generalized the image formation model further to I = ∫ 1 0 Fτ ·Mτ dτ + ( 1− ∫ 1 0 Mτ dτ ) ·B , ( 3 ) where the sharp object appearance Fτ is defined at any real-valued time point τ as an output of their encoder-decoder network architecture . The integration bounds correspond to the normalized image exposure time with start and end times 0 and 1 , respectively . In ( 3 ) , the 2D object motion is directly encoded in the masks Mτ . The network is trained end-to-end on a synthetic dataset with various moving 3D objects . DeFMO generalizes well to real-world data , but the outputs are still just 2D projections of the moving object . There is no guarantee that they correspond to the object ’ s actual 3D motion and shape , and there is no straightforward way to estimate the 3D shape from 2D DeFMO outputs . The 3D motion is not estimated either . 3D shape from sharp image . Estimating 3D shape from a single sharp image is a hot topic in computer vision . Recent methods are based on differentiable or neural rendering [ 24 , 25 , 26 , 27 ] and Neural Radiance Fields ( NeRF ) [ 28 , 29 , 30 ] . Many of those methods heavily depend on shape priors and/or large datasets of training images ( of an object class [ 26 , 27 , 30 , 24 , 25 ] , sometimes even multiple views of the same object [ 28 , 29 ] ) . Over the last 30 years , a variety of methods have been proposed for monocular 3D shape estimation from other clues such as Shape from Shading [ 31 ] , Shape from Texture [ 32 ] , Shape from Focus [ 33 ] , Shape from Defocus [ 34 ] , Shape from Shadow [ 35 ] , Shape from Specularities [ 36 ] , and others . We augment this family of `` Shape from X '' methods with a new Shape from Blur ( SfB ) . To some degree , SfB can be related to Shape from Motion ( SfM ) [ 37 ] and Shape from Silhouettes [ 38 ] , but in our case , the 2D image sequence is collapsed and averaged into a single image by motion blur . In contrast to all previous FMO deblurring methods , we explicitly estimate textured 3D shape and 3D motion of such objects . In contrast to other 3D shape estimation methods , we estimate the 3D shape of an object with significant motion blur from a single image ( plus the background ) . 3 Method . The input to our method is a single RGB image I of a significantly blurred fast moving object . Additionally , an approximate image of the background B without FMOs is given . This can be estimated as a median of consecutive video frames or as a static image without the object . Hence , the assumption of known background is not very restrictive and also common among all FMO-related methods [ 10 , 18 , 23 , 11 ] . Our method outputs a textured 3D mesh Θ of the object and its sub-frame motion , represented as its initial 6-DoF pose { r , t } , and the 3D translation ∆t and 3D rotation ∆r it undergoes during the image exposure time . The key idea of our method is to reconstruct the textured 3D shape and motion parameters that best explain the input image . Concretely , we render the reconstructed object with its motion blur and define a loss comparing the resulting image with the input ( Fig . 2 ) . We then find the texture , shape , and motion parameters that minimize this loss . Shape representation . We represent the shape and appearance of an object by a textured triangular mesh Θ , which consists of a set of vertices , a set of faces , and a texture map with a given texture mapping . The mesh topology ( the set of faces and their connectivity ) is fixed . Our method estimates offsets from the initial vertex positions of a given prototype shape to deform it into the 3D shape of the object in the input image . We automatically select a prototype shape out of a given set based on the quality of the rendered image ( see Sec . 3.2 ) . Image formation model . We define two rendering functionsRF andRS that output the projected object appearance F = RF ( Θ ) and the object silhouette S = RS ( Θ ) as 2D images from a given mesh Θ . We use the Differentiable Interpolation-based Renderer ( DIB-R ) [ 24 ] to approximate functionsRF andRS in a differentiable manner . The camera is assumed to be static . Moreover , we define a mesh transformation functionM ( Θ , r , t ) that rotates mesh vertices around their center of mass by a quaternion-encoded 3D rotation r ∈ R4 and translates them by a 3D vector t ∈ R3 . Then , the generalized rendering-based image formation model Î ( Θ , r , ∆r , t , ∆t ; B ) is defined by Î ( · ) = ∫ 1 0 RF ( M ( Θ , r+τ ·∆r , t+τ ·∆t ) ) dτ+ ( 1− ∫ 1 0 RS ( M ( Θ , r+τ ·∆r , t+τ ·∆t ) ) dτ ) ·B ( 4 ) where time step τ = 0 corresponds to the initial object pose at the beginning of the camera exposure time , and τ = 1 corresponds to the final pose at the end . This image formation model generalizes all previous formation models for FMOs ( 1 ) , ( 2 ) , ( 3 ) . In goes beyond by explicitly modeling the object ’ s 3D shape and 3D motion . Thus , the parameters that describe the object and its motion are the motion representation { r , ∆r , t , ∆t } and Θ , which contains the mesh vertices offsets and the texture map .

Summary: This paper presents a method for 3D shape reconstruction of objects with large motion blurs in a video sequence. The authors propose to augment an off-the-shelf 2D shape-from-blur method, whose output is used to fit a 3D mesh shape as well as a linear 3D motion trajectory within a small timeframe. The motion blur is synthesized by averaging the shape & texture predictions within the timeframe for the image reconstruction loss, in addition to other regularization terms to keep the optimization well-behaved. Temporal super-resolution can also be achieved by synthesizing the image at a continuous timestamp. Experiments are benchmarked on 3 real-world datasets containing sequences of small objects and a synthetic dataset for benchmarking reconstruction accuracy.