Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects
Year: 2021

Abstract:
We address the novel task of jointly reconstructing the 3D shape, texture, and motion of an object from a single motion-blurred image. While previous approaches address the deblurring problem only in the 2D image domain, our proposed rigorous modeling of all object properties in the 3D domain enables the correct description of arbitrary object motion. This leads to significantly better image decomposition and sharper deblurring results. We model the observed appearance of a motion-blurred object as a combination of the background and a 3D object with constant translation and rotation. Our method minimizes a loss on reconstructing the input image via differentiable rendering with suitable regularizers. This enables estimating the textured 3D mesh of the blurred object with high fidelity. Our method substantially outperforms competing approaches on several benchmarks for fast moving objects deblurring. Qualitative results show that the reconstructed 3D mesh generates high-quality temporal super-resolution and novel views of the deblurred object.

heading: 1 Introduction
text: Motion blur is a common cause of degraded image quality. It can originate from camera motion, fast object motion, long exposure times due to low light settings, or a combination of these effects. In this paper, we focus on deblurring of fast moving objects (FMOs), which move over a distance larger than their size for the duration of the camera exposure of a single image. Since FMOs appear as a mixture of motion-blurred object texture and the background, our first goal is to decompose the combined appearance of the background and the one of the FMO in form of a matting mask [1]. Our main goal is to recover the shape, motion, and sharp texture of the FMO in order to explain its appearance in the input image in the best possible way (Fig. 1). The accurate reconstruction and motion estimation of FMOs enables to generate videos with temporal super-resolution, as well as FMO motion analysis and prediction. This is useful for a variety of applications like sports analysis (e.g. soccer, tennis, baseball), meteorites detection in astrophysics, detecting fast moving obstacles in front of a vehicle for driver alert or autonomous driving, and general video quality enhancement or compression. Our work could eventually make FMO tracking accessible to everyone using regular cameras rather than expensive high-speed cameras, or push the capabilities of current high-speed cameras to a new level, e.g. by capturing gun shots.
The majority of deblurring methods aim for the generic task of removing any kind of blur in images [2, 3, 4] or videos [5, 6, 7, 8, 9]. However, they perform poorly on deblurring of FMOs as
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
shown in comparisons to methods specialized for them [10, 11]. To the best of our knowledge, all generic and all FMO-specialized methods try to tackle the deblurring task only in the 2D image domain. This limits their ability to describe object rotations around any axis that does not correspond to the camera viewing direction, or any object translation along the viewing direction.
Our key idea is to model the object shape, motion, and texture in the 3D domain to better explain the input image as the projection and temporal aggregation of a continuously moving object. We recover motion as the initial 6-DoF pose of the object and the 3D translation and 3D rotation that it undergoes during the exposure time of the input image. Moreover, the proposed method is the first to estimate the 3D shape of motion-blurred objects. Although we are solving a more general and more challenging problem than previous methods, we achieve substantially better deblurring results due to the rigorous joint modeling of object shape, motion, and texture in 3D.
In summary, we make the following contributions: (1) We propose a novel fast moving object deblurring method that for the first time jointly estimates from a single input image the 3D shape, texture, and motion of an object (initial 6-DoF pose, 3D translation and 3D rotation). (2) We describe an effective pipeline that uses pure test-time optimization for the recovery of the object 3D shape, motion, and sharp appearance. To constrain the optimization, we propose several suitable regularizers.

heading: 2 Related work
text: Generic deblurring. Single-image deblurring is studied in [12, 2], and video deblurring in [9, 5, 6, 7, 8]. Some methods were specifically proposed for motion deblurring [13, 14, 15], but they generate only a single output image or a motion flow. A more difficult task of estimating the temporal super-resolution from a single image has been addressed in [3, 4, 16], but only for small motions. Below we review methods specialized to deblurring fast moving objects (FMOs).
Constant 2D object appearance. The image formation with motion blur of a FMO [10, 17] was introduced as I = H ∗ F + (1−H ∗M) ·B , (1) where the input image I is formed as a composition of two terms. The first term represents the blurred object as a convolution of the sharp object appearance F and the blur kernel H . The second term encodes the influence of the background B, which depends on the object matting mask M . This model assumes that the object appearance is constant throughout the whole exposure time. Tracking by Deblatting (TbD) [18, 19], from deblurring and matting, was proposed to estimate the unknown
sharp object model (F,M) and its motion represented by H . To achieve this, Kotera et al. [18] applied ADMM variable splitting with appearance total variation and blur kernel sparsity priors. Follow-up methods improve the precision of the 2D trajectory estimation [20, 21]. The special case of planar object with planar rotation is studied in [22].
Piece-wise constant 2D object appearance. The constant appearance model (1) was extended to piece-wise constant [23] to deal with small amounts of appearance changes over time. The appearance might change over time due to object translation in 3D, object rotation in 3D, view-specific changes due to illumination, specularities, shadows, and even object deformation, e.g. when the object bounces off the ground. The piece-wise constant model is defined as
I = ∑
τ
Hτ ∗ Fτ + ( 1− ∑
τ
Hτ ∗Mτ ) ·B , (2)
where the exposure time is split into several uniform time intervals τ (sub-frames). Each sub-frame has its own appearance model (Fτ ,Mτ ) and blur kernel Hτ , which are constant within the sub-frame. TbD-3D [23] proposed to estimate these piece-wise appearances by energy minimization with several priors, such as rotational symmetry, appearance total variation, and penalizing appearance from changing rapidly across neighboring sub-frames. The blur kernel is still estimated by deblatting [18] assuming the simple constant appearance model (1), and the sub-frame appearances are estimated in a second pass by splitting the trajectory into the desired number of pieces. Thus, TbD-3D performs well only with very small appearance changes and mostly spherical objects due to the rotational symmetry prior. For perfectly spherical objects, TbD-3D can also estimate 3D rotation by minimizing the sphere reprojection error. Then, the object-camera distance is approximated by the radius of estimated 2D masks. However, this method cannot generalize to other shapes.
2D projections of any object. DeFMO [11] was the first learning-based method for FMO deblurring. They generalized the image formation model further to
I =
∫ 1
0
Fτ ·Mτ dτ + ( 1− ∫ 1
0
Mτ dτ ) ·B , (3)
where the sharp object appearance Fτ is defined at any real-valued time point τ as an output of their encoder-decoder network architecture. The integration bounds correspond to the normalized image exposure time with start and end times 0 and 1, respectively. In (3), the 2D object motion is directly encoded in the masks Mτ . The network is trained end-to-end on a synthetic dataset with various moving 3D objects. DeFMO generalizes well to real-world data, but the outputs are still just 2D projections of the moving object. There is no guarantee that they correspond to the object’s actual 3D motion and shape, and there is no straightforward way to estimate the 3D shape from 2D DeFMO outputs. The 3D motion is not estimated either.
3D shape from sharp image. Estimating 3D shape from a single sharp image is a hot topic in computer vision. Recent methods are based on differentiable or neural rendering [24, 25, 26, 27] and Neural Radiance Fields (NeRF) [28, 29, 30]. Many of those methods heavily depend on shape priors and/or large datasets of training images (of an object class [26, 27, 30, 24, 25], sometimes even multiple views of the same object [28, 29]). Over the last 30 years, a variety of methods have been proposed for monocular 3D shape estimation from other clues such as Shape from Shading [31], Shape from Texture [32], Shape from Focus [33], Shape from Defocus [34], Shape from Shadow [35], Shape from Specularities [36], and others. We augment this family of "Shape from X" methods with a new Shape from Blur (SfB). To some degree, SfB can be related to Shape from Motion (SfM) [37] and Shape from Silhouettes [38], but in our case, the 2D image sequence is collapsed and averaged into a single image by motion blur. In contrast to all previous FMO deblurring methods, we explicitly estimate textured 3D shape and 3D motion of such objects. In contrast to other 3D shape estimation methods, we estimate the 3D shape of an object with significant motion blur from a single image (plus the background).

heading: 3 Method
text: The input to our method is a single RGB image I of a significantly blurred fast moving object. Additionally, an approximate image of the background B without FMOs is given. This can be estimated as a median of consecutive video frames or as a static image without the object. Hence, the
assumption of known background is not very restrictive and also common among all FMO-related methods [10, 18, 23, 11]. Our method outputs a textured 3D mesh Θ of the object and its sub-frame motion, represented as its initial 6-DoF pose {r, t}, and the 3D translation ∆t and 3D rotation ∆r it undergoes during the image exposure time. The key idea of our method is to reconstruct the textured 3D shape and motion parameters that best explain the input image. Concretely, we render the reconstructed object with its motion blur and define a loss comparing the resulting image with the input (Fig. 2). We then find the texture, shape, and motion parameters that minimize this loss.
Shape representation. We represent the shape and appearance of an object by a textured triangular mesh Θ, which consists of a set of vertices, a set of faces, and a texture map with a given texture mapping. The mesh topology (the set of faces and their connectivity) is fixed. Our method estimates offsets from the initial vertex positions of a given prototype shape to deform it into the 3D shape of the object in the input image. We automatically select a prototype shape out of a given set based on the quality of the rendered image (see Sec. 3.2).
Image formation model. We define two rendering functionsRF andRS that output the projected object appearance F = RF (Θ) and the object silhouette S = RS(Θ) as 2D images from a given mesh Θ. We use the Differentiable Interpolation-based Renderer (DIB-R) [24] to approximate functionsRF andRS in a differentiable manner. The camera is assumed to be static. Moreover, we define a mesh transformation functionM(Θ, r, t) that rotates mesh vertices around their center of mass by a quaternion-encoded 3D rotation r ∈ R4 and translates them by a 3D vector t ∈ R3. Then, the generalized rendering-based image formation model Î(Θ, r,∆r, t,∆t;B) is defined by
Î(·) = ∫ 1
0
RF ( M(Θ, r+τ ·∆r, t+τ ·∆t) ) dτ+ ( 1− ∫ 1
0
RS ( M(Θ, r+τ ·∆r, t+τ ·∆t) ) dτ ) ·B (4)
where time step τ = 0 corresponds to the initial object pose at the beginning of the camera exposure time, and τ = 1 corresponds to the final pose at the end. This image formation model generalizes all previous formation models for FMOs (1), (2), (3). In goes beyond by explicitly modeling the object’s 3D shape and 3D motion. Thus, the parameters that describe the object and its motion are the motion representation {r,∆r, t,∆t} and Θ, which contains the mesh vertices offsets and the texture map.

heading: 3.1 Loss terms
text: Image formation loss. The main component of our method is the image formation loss that measures the input image reconstruction according to (4). It is the difference between the observed input image and its reconstruction by the rendering-based formation model:
LI(Θ, r,∆r, t,∆t; I,B) = ‖I − Î(Θ, r,∆r, t,∆t;B)‖1 . (5)
We noticed that (5) is directly related to the quality of reconstruction and deblurring as measured by the evaluation metrics PSNR and SSIM on the FMO deblurring benchmark [11] (see supplementary). However, directly minimizing the loss (5) is challenging since it is under-constrained and with many local minima, as observed experimentally. Therefore, additional priors and regularizers are vital.
Silhouette consistency loss. Another important prior is that the rendered silhouettes should stay close to sub-frame masks Mτ estimated by DeFMO [11]. This guides the 3D shape estimation
with the approximate object location in the image, which helps especially in the early steps of the optimization process. The silhouette consistency is defined as
LS(Θ, r,∆r, t,∆t) = 1− ∫ 1
0
IoU ( Mτ ,RS ( M(Θ, r + τ ·∆r, t + τ ·∆t) )) dτ , (6)
where the Intersection over Union (IoU) is computed between the mesh silhouettes estimated by our method and the input DeFMO masks.
Laplacian shape regularization loss. Due to the 3D-to-2D projection, many meshes generate the same input image, especially in the mesh areas not visible in the image (‘back sides’ of the object). Thus, in order to favor smooth meshes, we add a commonly used [24, 26, 25] Laplacian shape regularization LL(Θ). Texture smoothness loss. We favor smooth piecewise-constant textures by adding a total variation loss LT (Θ) on the estimated texture map, which is a sparse gradient prior [39], computed as the mean absolute difference between neighboring pixels in both image directions. Thanks to this term, the texture in unobserved mesh parts is a smooth transition between the edges of the observed parts. Without LT (Θ), our method produces random noise pixels in the unobserved texture parts, which degrade the quality of novel view synthesis.
Joint loss. The joint loss is a weighed sum of the image formation loss (5), the silhouette consistency (6), Laplacian shape regularization, and texture smoothness:
L = LI(Θ, r,∆r, t,∆t; I,B) + LS(Θ, r,∆r, t,∆t) + LT (Θ) + λL · LL(Θ) . (7)
In Sec. 3.2, we infer the parameters {Θ, r,∆r, t,∆t} describing the object texture, shape, and motion by test-time optimization on the input image I and background B.
Technical details. The weight of the Laplacian shape regularization is set to λL = 1000. There are no weights for other terms since the default unit weight produced good results. We use the Kaolin [40] implementation of DIB-R [24], and set the smoothness term for soft mask rasterization to 7000−1. We use a median of the previous five frames in a video as an approximation of the background.

heading: 3.2 Loss optimization
text: We optimize the joint loss (7) using ADAM [41] with the fixed learning rate 0.1 for 500 iterations in the PyTorch [42] framework. This is made possible by using a differentiable renderer, which enables to compute gradients of the rendered image w.r.t. the object parameters. Optimization takes about 60 seconds on a single 8 GB Nvidia GTX 1080 GPU (for 500 iterations of a single prototype mesh with 1212 vertices, 2420 faces, 100× 100 texture map, 8 time steps to discretize the intergral, and input image of 240× 320 pixels). Prototypes. We provide the method with a set of three prototypes. The best prototype is automatically selected based on the lowest value of the image formation loss (5) after optimization starting from each prototype in turn. The first prototype is a sphere with 1212 vertices, 2420 faces, and simple spherical projection texture mapping. The second prototype is a bigger sphere with 1538 vertices, 3072 faces, and with more sophisticated Voronoi texture mapping [43]. The third prototype is a torus with a similar number of vertices and faces and the same Voronoi mapping. We used two spheres with different number of vertices to include more diversity in the complexity of the genus 0 shape. More vertices allow modeling more local shape details, e.g. concavities.
Normalization and initialization. The mesh vertices are normalized to zero center of mass and unit variance, maintaining the normalization throughout the whole optimization process. Normalization to unit variance is needed to keep the mesh in the canonical space. Without such normalization, it would be ambiguous whether the mesh vertices should be moved or the initial pose {r, t} changed. Moreover, this normalization makes it easier to keep the object projection within the image bounds.
Texture values are normalized by Sigmoid activation. Translation vectors t and t + ∆t are estimated as a hyperbolic tangent (tanh) activation with normalization such that the 2D object projection is always visible in the image, i.e. (−1,−1) is the bottom left corner of the image, (1, 1) is the top right corner, a z coordinate of −1 indicates an object covering the whole field-of-view (FOV), and z = 1 indicates covering 5% of FOV. Rotations r,∆r are represented by quaternions, and ∆r is capped to represent at most 120 degrees total rotation. The mesh is initialized with a uniform grey texture, zero vertex offsets, zero rotation, zero translation, and placed in the center of the image. Experimentally, initializing the texture by projecting the blurry input image directly onto the mesh led to inferior performance. The reason might be that blurry object appearance is a local optimum for the texture, making it more difficult for the optimization process to converge to the sharp object appearance.
In sum, the proposed method estimates the object’s textured mesh Θ and its motion {r,∆r, t,∆t} by test-time optimization. The input image is explained by the object parameters via minimizing the joint loss (7) that consists of the image formation loss (5) and suitable regularizers.

heading: 4 Evaluation
text: Datasets. We evaluate our method on four datasets. Three real-world datasets from the FMO deblurring benchmark [11]: (1) TbD [18], with 12 sequences (471 frames total); (2) TbD-3D [23], with 10 sequences (516 frames); (3) Falling Objects [22], with 6 sequences (94 frames); (4) and a synthetic dataset from [11]. The easiest dataset is TbD [18] with uniformly colored, mostly spherical objects. TbD-3D [23] is more difficult and contains objects with complex texture and significant 3D rotation, but the objects are still mostly spherical. Falling Objects [22] is the most challenging dataset, with objects of complex 3D shape (e.g. key, pen, marker) and significant rotation. The ground truth for all three real-world datasets includes the high-speed camera footage at 8× higher frame rate, as well as the sub-frame 2D object locations and 2D masks. We use the synthetic dataset to evaluate the 3D shape Θ, 3D translation ∆t, and 3D rotation ∆r estimated by our method, as there are no real-world datasets of blurred objects with 3D models annotations. The dataset is created by applying the rendering-based image formation model (4) to synthetically moving 3D ShapeNet [44]
objects. The backgrounds are randomly sampled from sequences in the VOT dataset [45]. Rotation and translation are generated randomly, with an average translation of 3× the object size and an average rotation of 100◦. This way, we generate 10 images, one for each of 10 different objects.
Evaluation metrics. We evaluate the deblurring accuracy by comparing the sub-frame object reconstruction to high-speed camera frames. We use Trajectory-IoU (TIoU) [18], which averages the standard IoU between the estimated silhouette and the ground truth mask along the ground truth trajectory. For evaluating sub-frame appearance, we generate a temporal super-resolution image sequence by applying the rendering-based image formation model (4) at a series of time sub-intervals (changing the integration bounds) using the object parameters {Θ, r,∆r, t,∆t} estimated by our method. We then compare this generated sequence to the ground truth high-speed frames by the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) metrics.
Next, we evaluate the error in estimating the 3D translation ∆t and 3D rotation ∆r that the object undergoes during the camera exposure time. The translation error is measured as ∆t = ‖∆t − ∆tGT‖2, and the rotation error ∆r is the angle between ∆r and ∆rGT (GT denotes ground truth values). Note that the canonical mesh representation of ShapeNet objects is different from the one estimated by our method, preventing us from evaluating the estimate of the initial 6-DoF pose {r, t}. To measure the 3D shape reconstruction accuracy, we align the estimated Θ and ground truth ΘGT meshes and compute the average bidirectional distance between them as Θ = 1/2(D(Θ,ΘGT) + D(ΘGT,Θ)), where D(Θ1,Θ2) is the average distance of each vertex of Θ1 to the closest face of Θ2. The meshes are rescaled to tightly fit a unit cube, thus a unit of Θ and t is one object size (diameter).

heading: 4.1 Comparison to previous methods on TbD, TbD-3D, and Falling Objects datasets
text: We compare to two generic deblurring methods, DeblurGAN-v2 [2] and Jin et al. [3]. We also compare to methods specialized for FMO deblurring, which are the TbD method [18], TbD-3D method [23], and the state-of-the-art DeFMO [11].
Our SfB method outperforms the above methods on all metrics on the two most challenging datasets Falling Objects and TbD-3D, setting the new state-of-the-art (Table 1). On the easier TbD dataset, SfB achieves the highest TIoU and PSNR, but marginally lower SSIM than TbD-3D, which is specifically designed for uniformly colored spherical objects. We show qualitative results in Fig. 3 and Fig. 4.

heading: 4.2 Ablation study on Falling Objects and synthetic datasets
text: The first block in the ablation study (Table 2) shows that omitting the silhouette consistency loss LS (6) degrades performance by a wide margin, highlighting its importance. We also experiment with an alternative form of the silhouette consistency loss, which uses simple background subtraction instead of the DeFMO masks within (6). More precisely, we replace Mτ with the difference D between the input image and the background, thresholded at 0.1 (details in supplementary). Using LD is better than completely omitting any silhouette consistency, but performs worse than LS , since DeFMO localizes the object with the sub-frame precision, thereby providing a strong signal to our method. We also tried another loss LF that favors the textures rendered byRF to be similar (in terms of L2 loss) to the sub-frame appearances Fτ estimated by DeFMO. As shown in Table 2, this loss
degrades the performance since DeFMO appearances are still blurry, noisy, and the set of Fτ does not correspond to a consistent 3D object (Fig. 3).
The second block of Table 2 shows the improvements of the Laplacian shape regularization and texture smoothness losses on the Falling Objects dataset. However, since the shapes and textures in the synthetic dataset are significantly more complex (e.g. guitar, jar, skateboard, sofa, bottle, pillow), these losses increase the 3D errors by a small margin.
The third block demonstrates the benefits of using more prototypes. In order to test the robustness of the automatic prototype selection, we also include additional distractor prototypes (teapot, teddy, bunny, cow) that are not similar to other objects in the datasets. Adding them changes the performance only marginally since they are rarely selected, demonstrating robustness. Besides, the first two rows of this block also show that ignoring the rotation leads to a drop in performance.
The last block evaluates step count used for discretizing the integral in the rendering-based image formation model (4). The performance increases with the number of steps, until it saturates at around 5× more steps than the number of ground truth high-speed frames available for each image (i.e. 8). According to the ablation study on the Falling Objects dataset (Table 2), the best performing version of SfB is using all loss terms (including DeFMO masks in LS), 7 prototypes (including 4 distractors), and 5× more integration steps than the desired temporal super-resolution output. This version takes 30 minutes optimization time due to the high number of prototypes and time steps, which improve performance only marginally. While we can run our method in a configuration taking only a few minutes, optimization time can still be seen as a limitation. We believe that faster runtime could be achieved by avoiding unnecessary computations and iterations, and leave this as future work.

heading: 4.3 Applications to super-resolution and novel view synthesis
text: The main application of SfB is temporal super-resolution. For videos of motion-blurred objects, SfB can generate a video of arbitrary higher frame rates by correctly modeling the object’s 3D shape and motion (Fig. 3). The modeling of textured 3D shape allows to generate novel views of the fast moving object (Fig. 6). SfB correctly recovers the 3D shape and deblurred texture of various moving objects captured by a mobile device (Fig. 4, boots, tape roll) or found on YouTube (Fig. 4, tennis ball).
3D object rotation estimation has applications in sports analysis, e.g. in professional tennis games. For known intrinsic camera parameters, velocity and rotation can be estimated in real-world units.

heading: 4.4 Limitations
text: Since we assume a static camera, the method is not designed to deal with blur induced by a moving or shaking camera. Experimentally, we observed that the method is robust to small amounts of moving camera blur. Among other limitations are long optimization time, rigid body assumptions, and inconsistent estimation of unobserved object parts.
The proposed image formation model does not model any type of image noise, e.g. additive image noise, JPEG artifacts, compression, and others. In general, these factors can deteriorate the performance of the method (Fig. 5). However, the minimization of the reprojection error suppresses noise, and even the basic image formation model without the image noise still leads to satisfactory results in many real-world scenarios, as shown in the experimental section.

heading: 5 Conclusions and future work
text: We introduced and tackled a novel task to estimate the textured 3D shape and motion of fast moving objects given a single image with a known background. The proposed method is the first solution to this problem, and also sets a new state of the art on 2D fast moving object deblurring. Since the shape estimator deforms the (often spherical) prototype shape, the shape often remains unchanged along unobserved directions (Fig. 6, pen). To address some of the limitations, future work includes adding shape priors, such as symmetry or data-driven priors, and more powerful shape representations for expressing arbitrary shape topologies. The differentiable SfB loss can be incorporated into deep learning pipelines to benefit from large training datasets. SfB can be generalized to optimize over multiple video frames, rather than just one. For instance, when optimizing the textured 3D shape for several consecutive frames, the object becomes visible from more viewpoints providing more constraints and fewer ambiguities in unobserved parts.
Acknowledgments and Disclosure of Funding
This work was supported by a Google Focused Research Award, Innosuisse grant No. 34475.1 IP-ICT, and a research grant by the International Federation of Association Football (FIFA).

Review summaries:
This paper introduces an approach for predicting the textured 3D shape and motion from blurred video frames. The approach is based on the differentiable rendering that can predict shapes and image formations that can be regarded as a combination of background and foreground scenes. The paper shows interesting results that can reasonably recover shapes, motion, and a deblurred texture map.

This paper presents SfB, a novel method of handling a single RGB image of a significantly blurred fast moving object, by recovering the object's shape, texture, and trajectory, what are optimized towards a joint loss of image consistencies, via a differentiable render. The method can be applied to a variant of tasks of de-blurring, shape reconstruction, super resolution and view synthesis. The paper reports state-of-the-art results on a variant of real-world and synthetic datasets, both quantitatively and qualitatively.

The submitted paper focuses on the shape, texture, and motion recovery of a single fast moving object from two input images from a static camera - a clean background image, and a blurred image due to fast object motions. The authors tackle the problem using differential rendering and iterative optimization at test time. Specifically, the template mesh ("prototype"), the texture map, the initial translation and rotation, as well as the constant motion speed are jointly optimized in order to minimize a combination of loss functions, with the image formation loss being the key contribution. The method is demonstrated on both synthetic dataset and relatively simple real datasets. 

This paper presents a method for 3D shape reconstruction of objects with large motion blurs in a video sequence. The authors propose to augment an off-the-shelf 2D shape-from-blur method, whose output is used to fit a 3D mesh shape as well as a linear 3D motion trajectory within a small timeframe. The motion blur is synthesized by averaging the shape & texture predictions within the timeframe for the image reconstruction loss, in addition to other regularization terms to keep the optimization well-behaved. Temporal super-resolution can also be achieved by synthesizing the image at a continuous timestamp. Experiments are benchmarked on 3 real-world datasets containing sequences of small objects and a synthetic dataset for benchmarking reconstruction accuracy.