Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects

1 Introduction . Motion blur is a common cause of degraded image quality . It can originate from camera motion , fast object motion , long exposure times due to low light settings , or a combination of these effects . In this paper , we focus on deblurring of fast moving objects ( FMOs ) , which move over a distance larger than their size for the duration of the camera exposure of a single image . Since FMOs appear as a mixture of motion-blurred object texture and the background , our first goal is to decompose the combined appearance of the background and the one of the FMO in form of a matting mask [ 1 ] . Our main goal is to recover the shape , motion , and sharp texture of the FMO in order to explain its appearance in the input image in the best possible way ( Fig . 1 ) . The accurate reconstruction and motion estimation of FMOs enables to generate videos with temporal super-resolution , as well as FMO motion analysis and prediction . This is useful for a variety of applications like sports analysis ( e.g . soccer , tennis , baseball ) , meteorites detection in astrophysics , detecting fast moving obstacles in front of a vehicle for driver alert or autonomous driving , and general video quality enhancement or compression . Our work could eventually make FMO tracking accessible to everyone using regular cameras rather than expensive high-speed cameras , or push the capabilities of current high-speed cameras to a new level , e.g . by capturing gun shots . The majority of deblurring methods aim for the generic task of removing any kind of blur in images [ 2 , 3 , 4 ] or videos [ 5 , 6 , 7 , 8 , 9 ] . However , they perform poorly on deblurring of FMOs as 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . shown in comparisons to methods specialized for them [ 10 , 11 ] . To the best of our knowledge , all generic and all FMO-specialized methods try to tackle the deblurring task only in the 2D image domain . This limits their ability to describe object rotations around any axis that does not correspond to the camera viewing direction , or any object translation along the viewing direction . Our key idea is to model the object shape , motion , and texture in the 3D domain to better explain the input image as the projection and temporal aggregation of a continuously moving object . We recover motion as the initial 6-DoF pose of the object and the 3D translation and 3D rotation that it undergoes during the exposure time of the input image . Moreover , the proposed method is the first to estimate the 3D shape of motion-blurred objects . Although we are solving a more general and more challenging problem than previous methods , we achieve substantially better deblurring results due to the rigorous joint modeling of object shape , motion , and texture in 3D . In summary , we make the following contributions : ( 1 ) We propose a novel fast moving object deblurring method that for the first time jointly estimates from a single input image the 3D shape , texture , and motion of an object ( initial 6-DoF pose , 3D translation and 3D rotation ) . ( 2 ) We describe an effective pipeline that uses pure test-time optimization for the recovery of the object 3D shape , motion , and sharp appearance . To constrain the optimization , we propose several suitable regularizers . 2 Related work . Generic deblurring . Single-image deblurring is studied in [ 12 , 2 ] , and video deblurring in [ 9 , 5 , 6 , 7 , 8 ] . Some methods were specifically proposed for motion deblurring [ 13 , 14 , 15 ] , but they generate only a single output image or a motion flow . A more difficult task of estimating the temporal super-resolution from a single image has been addressed in [ 3 , 4 , 16 ] , but only for small motions . Below we review methods specialized to deblurring fast moving objects ( FMOs ) . Constant 2D object appearance . The image formation with motion blur of a FMO [ 10 , 17 ] was introduced as I = H ∗ F + ( 1−H ∗M ) ·B , ( 1 ) where the input image I is formed as a composition of two terms . The first term represents the blurred object as a convolution of the sharp object appearance F and the blur kernel H . The second term encodes the influence of the background B , which depends on the object matting mask M . This model assumes that the object appearance is constant throughout the whole exposure time . Tracking by Deblatting ( TbD ) [ 18 , 19 ] , from deblurring and matting , was proposed to estimate the unknown sharp object model ( F , M ) and its motion represented by H . To achieve this , Kotera et al . [ 18 ] applied ADMM variable splitting with appearance total variation and blur kernel sparsity priors . Follow-up methods improve the precision of the 2D trajectory estimation [ 20 , 21 ] . The special case of planar object with planar rotation is studied in [ 22 ] . Piece-wise constant 2D object appearance . The constant appearance model ( 1 ) was extended to piece-wise constant [ 23 ] to deal with small amounts of appearance changes over time . The appearance might change over time due to object translation in 3D , object rotation in 3D , view-specific changes due to illumination , specularities , shadows , and even object deformation , e.g . when the object bounces off the ground . The piece-wise constant model is defined as I = ∑ τ Hτ ∗ Fτ + ( 1− ∑ τ Hτ ∗Mτ ) ·B , ( 2 ) where the exposure time is split into several uniform time intervals τ ( sub-frames ) . Each sub-frame has its own appearance model ( Fτ , Mτ ) and blur kernel Hτ , which are constant within the sub-frame . TbD-3D [ 23 ] proposed to estimate these piece-wise appearances by energy minimization with several priors , such as rotational symmetry , appearance total variation , and penalizing appearance from changing rapidly across neighboring sub-frames . The blur kernel is still estimated by deblatting [ 18 ] assuming the simple constant appearance model ( 1 ) , and the sub-frame appearances are estimated in a second pass by splitting the trajectory into the desired number of pieces . Thus , TbD-3D performs well only with very small appearance changes and mostly spherical objects due to the rotational symmetry prior . For perfectly spherical objects , TbD-3D can also estimate 3D rotation by minimizing the sphere reprojection error . Then , the object-camera distance is approximated by the radius of estimated 2D masks . However , this method can not generalize to other shapes . 2D projections of any object . DeFMO [ 11 ] was the first learning-based method for FMO deblurring . They generalized the image formation model further to I = ∫ 1 0 Fτ ·Mτ dτ + ( 1− ∫ 1 0 Mτ dτ ) ·B , ( 3 ) where the sharp object appearance Fτ is defined at any real-valued time point τ as an output of their encoder-decoder network architecture . The integration bounds correspond to the normalized image exposure time with start and end times 0 and 1 , respectively . In ( 3 ) , the 2D object motion is directly encoded in the masks Mτ . The network is trained end-to-end on a synthetic dataset with various moving 3D objects . DeFMO generalizes well to real-world data , but the outputs are still just 2D projections of the moving object . There is no guarantee that they correspond to the object ’ s actual 3D motion and shape , and there is no straightforward way to estimate the 3D shape from 2D DeFMO outputs . The 3D motion is not estimated either . 3D shape from sharp image . Estimating 3D shape from a single sharp image is a hot topic in computer vision . Recent methods are based on differentiable or neural rendering [ 24 , 25 , 26 , 27 ] and Neural Radiance Fields ( NeRF ) [ 28 , 29 , 30 ] . Many of those methods heavily depend on shape priors and/or large datasets of training images ( of an object class [ 26 , 27 , 30 , 24 , 25 ] , sometimes even multiple views of the same object [ 28 , 29 ] ) . Over the last 30 years , a variety of methods have been proposed for monocular 3D shape estimation from other clues such as Shape from Shading [ 31 ] , Shape from Texture [ 32 ] , Shape from Focus [ 33 ] , Shape from Defocus [ 34 ] , Shape from Shadow [ 35 ] , Shape from Specularities [ 36 ] , and others . We augment this family of `` Shape from X '' methods with a new Shape from Blur ( SfB ) . To some degree , SfB can be related to Shape from Motion ( SfM ) [ 37 ] and Shape from Silhouettes [ 38 ] , but in our case , the 2D image sequence is collapsed and averaged into a single image by motion blur . In contrast to all previous FMO deblurring methods , we explicitly estimate textured 3D shape and 3D motion of such objects . In contrast to other 3D shape estimation methods , we estimate the 3D shape of an object with significant motion blur from a single image ( plus the background ) . 3 Method . The input to our method is a single RGB image I of a significantly blurred fast moving object . Additionally , an approximate image of the background B without FMOs is given . This can be estimated as a median of consecutive video frames or as a static image without the object . Hence , the assumption of known background is not very restrictive and also common among all FMO-related methods [ 10 , 18 , 23 , 11 ] . Our method outputs a textured 3D mesh Θ of the object and its sub-frame motion , represented as its initial 6-DoF pose { r , t } , and the 3D translation ∆t and 3D rotation ∆r it undergoes during the image exposure time . The key idea of our method is to reconstruct the textured 3D shape and motion parameters that best explain the input image . Concretely , we render the reconstructed object with its motion blur and define a loss comparing the resulting image with the input ( Fig . 2 ) . We then find the texture , shape , and motion parameters that minimize this loss . Shape representation . We represent the shape and appearance of an object by a textured triangular mesh Θ , which consists of a set of vertices , a set of faces , and a texture map with a given texture mapping . The mesh topology ( the set of faces and their connectivity ) is fixed . Our method estimates offsets from the initial vertex positions of a given prototype shape to deform it into the 3D shape of the object in the input image . We automatically select a prototype shape out of a given set based on the quality of the rendered image ( see Sec . 3.2 ) . Image formation model . We define two rendering functionsRF andRS that output the projected object appearance F = RF ( Θ ) and the object silhouette S = RS ( Θ ) as 2D images from a given mesh Θ . We use the Differentiable Interpolation-based Renderer ( DIB-R ) [ 24 ] to approximate functionsRF andRS in a differentiable manner . The camera is assumed to be static . Moreover , we define a mesh transformation functionM ( Θ , r , t ) that rotates mesh vertices around their center of mass by a quaternion-encoded 3D rotation r ∈ R4 and translates them by a 3D vector t ∈ R3 . Then , the generalized rendering-based image formation model Î ( Θ , r , ∆r , t , ∆t ; B ) is defined by Î ( · ) = ∫ 1 0 RF ( M ( Θ , r+τ ·∆r , t+τ ·∆t ) ) dτ+ ( 1− ∫ 1 0 RS ( M ( Θ , r+τ ·∆r , t+τ ·∆t ) ) dτ ) ·B ( 4 ) where time step τ = 0 corresponds to the initial object pose at the beginning of the camera exposure time , and τ = 1 corresponds to the final pose at the end . This image formation model generalizes all previous formation models for FMOs ( 1 ) , ( 2 ) , ( 3 ) . In goes beyond by explicitly modeling the object ’ s 3D shape and 3D motion . Thus , the parameters that describe the object and its motion are the motion representation { r , ∆r , t , ∆t } and Θ , which contains the mesh vertices offsets and the texture map .

Summary: This paper introduces an approach for predicting the textured 3D shape and motion from blurred video frames. The approach is based on the differentiable rendering that can predict shapes and image formations that can be regarded as a combination of background and foreground scenes. The paper shows interesting results that can reasonably recover shapes, motion, and a deblurred texture map.