Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects
Authors: Denys Rozumnyi
Martin R. Oswald
Vittorio Ferrari
Marc Pollefeys
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3d shape reconstruction
deblurring
fast moving objects
motion blur
temporal super-resolution
test-time optimization
shape from blur

Abstract: We address the novel task of jointly reconstructing the 3D shape, texture, and motion of an object from a single motion-blurred image. While previous approaches address the deblurring problem only in the 2D image domain, our proposed rigorous modeling of all object properties in the 3D domain enables the correct description of arbitrary object motion. This leads to significantly better image decomposition and sharper deblurring results. We model the observed appearance of a motion-blurred object as a combination of the background and a 3D object with constant translation and rotation. Our method minimizes a loss on reconstructing the input image via differentiable rendering with suitable regularizers. This enables estimating the textured 3D mesh of the blurred object with high fidelity. Our method substantially outperforms competing approaches on several benchmarks for fast moving objects deblurring. Qualitative results show that the reconstructed 3D mesh generates high-quality temporal super-resolution and novel views of the deblurred object.

Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects
Denys Rozumnyi
Martin R. Oswald 1.2
Department of Computer Science ETH Zurich, Switzerland
University of Amsterdam
Netherlands
@inf www...eeeecmmmm ch
martin, oswald@inf ethz. ch
Vittorio Ferrari
Mare Pollefeys
Google Research
Department of Computer Science
Zurich, Switzerland vittoferrar1@@goggee.co
ETH Zurich, Switzerland
marc marc.polleffge pol..eeeeeeeeenn ethz. ch
Abstract
We address the novel task of jointly reconstructing the 3D shape, texture, and motion of an object from single motion- address the deblurring problem only in the 2D image domain, our proposed rigorous modeling of all object properties in the 3D domain enables the correct description of arbitrary object motion. This leads to significantly better image decomposition and sharper deblurring results. We model the observed appearance of noturn-bbluree object as combination of the background and a 3D object with constant translation and rotation. Our method minimizes a loss on reconstructing the input image via differentiable rendering with suitable regularizers. This enables estimating the textured 3D mesh of the blurred object with high fidelity. Our method substantially outperforms competing approaches on several benchmarks for fast moving objects deblurring. Qualitative results show that the reconstructed 3D mesh generates high-quality temporal super-resolution and novel views of the deblurred object.
image. While previous approaches
Introduction
Motion blur a common cause of degraded image quality. It can originate from camera motion, fast object motion, long exposure times due to low light settings, combination of these effects. In this paper, we focus on deblurring of fast moving objects (FMOs), which move over a distance larger than their size for the duration of the camera exposure of single image. Since FMOs appear of motion-blureed object texture and the background, our first goal is to decompose the combined appearance of the background and the one of the FMO in form of a matting mask (I] Our main goal is to recover the shape, motion, and sharp texture of the FMO in order to explain its appearance in the input image in the best possible way (Fig. T.. The accurate reconstruction and motion estimation of FMOs enables to generate videos with temporal super-resolution, as well as FMO motion analysis and prediction. This is useful for variety of applications like sports analysis (c.g. soccer, tennis, baseball), metcorites detection in astrophysics, detecting fast moving obstacles in front of vehicle for driver alert or autonomous driving, and general video quality enhancement or compression. Our work could eventually make FMO tracking accessible to everyone using regular cameras rather than expensive high-speed cameras, or push the capabilities of current speh-speed cameras new level, e.g. by capturing gun shots.
mixture
The majority of deblurring methods aim for the generic task of removing any kind of blur in images or videos 6 7 9.. However, they perform poorly on deblurring of FMOs as
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Pea volleyball
Inputs
Temporal super resolutin and aub-frame decomposition Novel views
Geo.
GT
Figure Temporal super-resolution and deblurring with 3D reconstruction of fast moving objects, given single image. Solely from the inputs on the left (clean background B, blurry image I), the proposed Shape-from-Blur (SfB) method reconstructs high-quality textured 3D shape and its motion, by minimizing the input image reconstruction loss with suitable regularizers. Our method enables generate temporal super-resolution from single image, as well synthesizing novel views of the reconstructed shape. The ground truth (GT) on the right shows the corresponding high-speed camera footage (rows 1-4) or picture of the object without motion blur (rows 5-6).
shown in comparisons to methods specialized for them [10 [1]. To the best of our knowledge, all generic and FMO-specialized methods try to tackle the deblurring task only in the 2D image domain. This limits their ability to describe object rotations around any axis that does not correspond to the camera viewing direction, or any object translation along the viewing direction.
Our key idea is to model the object shape, motion, and texture in the 3D domain to better explain the input image as the projection and temporal aggregation of continuously moving object. We recover motion as the initial 6-DoF pose of the object and the 3D translation and 3D rotation that undergoes during the exposure time of the input image. Morcover, the proposed method the first to estimate the 3D shape of motion-blurred objects. Although we solving more general and more challenging problem than previous methods, we achieve substantially better deblurring results due to the rigorous joint modeling of object shape, motion, and texture in 3D.
In summary, we make the following contributions: (1) We propose novel fast moving object deblurring method that for the first time jointly estimates from a single input image the 3D shape, texture, and motion of an object (initial 6-DoF pose, 3D translation and 3D rotation).
(2) We describe an effective pipeline that uses pure test-time optimization for the recovery of the object 3D shape, motion, and sharp appearance. To constrain the optimization, we propose several suitable regularizers.
2 Related work
Generic deblurring. Single-image deblurring is studied in 112 2 and video deblurring in 9 6 8) Some methods were specifically proposed for motion deblurring [13 [4 15). but they
generate only a single output image or a motion flow. A more difficult task of estimating the temporal super-resolution from single image has been addressed in Below we review methods specialized to deblurring fast moving objects (FMOs).
16], but only for small motions.
Constant 2D object appearance. The image formation with motion blur of FMO 10 17 was introduced as
1
B
(1)
where the input image I is formed as a composition of two terms. The first term represents the blurred object as a convolution of the sharp object appearance F and the blur kernel H. The second term encodes the influence of the background B, which depends on the object matting mask M. This model assumes that the object appearance is constant throughout the whole exposure time. Tracking by Deblatting (TbD) [18 19), from deblurring and matting, was proposed to estimate the unknown

---

sharp object model (F.M) and its motion represented by H. To achieve this, Kotera et al. [8 applied ADMM variable splitting with appearance total variation and blur kernel sparsity priors.
Follow-up methods improve the precision of the 2D trajectory estimation 20 21
The special case
of planar object with planar rotation is studied in [22]
Piece-wise constant 2D object appearance. The constant appearance model |I was extended to piece-wise constant [23] to deal with small amounts of appearance changes over time. The appearance might change over time due to object translation in 3D, object rotation in 3D, view-specific changes due to illumination, specularities. shadows, and even object deformation, e.g. when the object bounces off the ground. The piece-wise constant model is defined as
1= H F (1 Σ Hr * Mr) B
where the exposure time is split into several uniform time intervals r (sub-frames). Each sub-frame has its own appearance model (Fr, M,) and blur kernel H,, which are constant within the sub-frame. TbD-3D (23) proposed to estimate these piece-wise appearances by energy minimization with several priors, such as rotational symmetry, appearance total variation, and penalizing appearance from changing rapidly across neighboring sub-frames. The blur kernel is still estimated by deblatting [[8] assuming the simple constant appearance model (T). and the sub-frame appearances are estimated in a second pass by splitting the trajectory into the desired number of pieces. Thus, TbD-3D performs well only with very small appearance changes and mostly spherical objects due to the rotational symmetry prior. For perfectly spherical objects, TbD-3D can also estimate 3D rotation by minimizing the sphere reprojection error. Then, the object-camera distance is approximated by the radius of estimated 2D masks. However, this method cannot generalize to other shapes.
2D projections of any object. DeFMO [IT] was the first dearning-based method for FMO deblurring. They generalized the image formation model further to
I 1- 1-f F. Mr dr + (1-So (M,MMM,r)] Mr dr) B
F.-M,
(3)
where the sharp object appearance F, is defined at any real-valued time point 7 an output of their encoder-decoder network architecture. The integration bounds correspond to the normalized image exposure time with start and end times 0 and 1, respectively. In (3). the 2D object motion is directly encoded in the masks M,. The network is trained end-to-end on a synthetic dataset with various moving 3D objects. DeFMO generalizes well to real-world data, but the outputs are still just 2D projections of the moving object. There is no guarantee that they correspond to the object's 's actual 3D motion and shape, and there is no straightforward way to estimate the 3D shape from 2D DeFMO outputs. The 3D motion is not estimated either.
3D shape from sharp image. Estimating 3D shape from single sharp image is hot topic in computer vision. Recent methods are based on differentiable or neural rendering 24
25
26 27
and Neural Radiance Fields (NeRF) 28
29
30 Many of those methods heavily depend
shape sometimes
priors and/or large datasets of training images (of an object class [26] 27
30
24
25),
even multiple views of the same object 228
29 Over the last 30 years, variety of methods have
been proposed for monocular 3D shape estimation from other clues such as Shape from Shading 31.
Shape from Texture [32], Shape from Focus (33), Shape from Defocus (34), Shape from Shadow [35]
Shape from Specularities [36], and others. We augment this family of "Shape from X" methods with new Shape from Blur (SIB). To some degree, SIB can be related Shape from Motion (SIM) 37
and Shape from Silhouettes (38], but in our case, the 2D image sequence is collapsed and averaged into a single image by motion blur. In contrast to all previous FMO deblurring methods, we explicitly estimate textured 3D shape and 3D motion of such objects. In contrast to other 3D shape estimation methods, we estimate the 3D shape of an object with significant motion blur from a single image (plus the background).
3 Method
The input to our method is single RGB image I of significantly blurred fast moving object. Additionally, an approximate image of the background B without FMOs given. This can be estimated median of consecutive video frames as static image without the object. Hence, the
3

---

Decp noural network
Oaject ieprsseration
Laplacian shape
Difirrmmaale renderer
2(8, r. 1111
foesttnction
misl Tramler
Iniial Retetion
Trandational Change
Recatind
Change
(Maah
Testural
smoothnees loss
Input
Appearance
dr
Background 20
DeFMO
Masio.M,
Sihouette los
Dicrete trre ommple from contiuue rornatzed time domsis rpst mage eqpbeure time
Figure Overview of our method. Given an input image and a clean background. together with the matting masks produced by DeFMO [I]] our method estimates the 3D shape, texture, and motion of the object. This achieved by jointly minimizing four suitable loss functions.
assumption known background not very restrictive and also common among all FMO-related methods [10
18 23 III. Our method outputs textured 3D mesh θ of the object and its sub-frame motion, represented as its initial 6-DoF pose (r, t). and the 3D translation At and 3D rotation Ar it undergoes during the image exposure time. The key idea of method is reconstruct the textured 3D shape and motion parameters that best explain the input image. Concretely, we render the reconstructed object with its motion blur and define loss comparing the resulting image with the input (Fig. 2
We then find the texture, shape, and motion parameters that minimize this loss.
Shape representation. We represent the shape and appearance of object by textured triangular mesh + which consists of a set of vertices, set of faces, and a texture map with given texture mapping. The mesh topology (the set of faces and their connectivity) is fixed. Our method estimates offsets from the initial vertex positions of given prototype shape to deform into the 3D shape of the object in the input image. We automatically select prototype shape out given based on the quality of the rendered image (see Sec. 3.2
Image formation model. We define two rendering functions R2 and Rg that output the projected object appearance F = (+) and the object silhouette S = (e) as 2D images from given mesh + We the Differentiable interpolation-based Renderer (DIB-R) [24] approximate functions R and Rg in a differentiable manner. The camera is assumed to be static. Moreover, we define a mesh transformation function M(A,r, t) that rotates mesh vertices around their center of mass by innterrioo-ennodde 3D rotation r E R and translates them by 3D vector t E Rd. Then, the generalized rendering-basee based image formation model Í( r, Ar, t, At; B) is defined by
I(-) = f.xR R (M r (+T- (1T (MM (erdrrrttt--)))) . (4)
where time step r = 0 corresponds to the initial object pose at the beginning of the camera exposure time, and r 1 corresponds to the final pose at the end. This image formation model generalizes all previous formation models for FMOs I.. (2) (3). In goes beyond by explicitly modeling the object's 3D shape and 31) motion. Thus, the parameters that deseribe the object and its motion are the motion
representation (r, Ar, t, At) and O, which contains the mesh vertices offsets and the texture map.
3.1 Loss terms
Image formation loss. The main component of our method is the image formation loss that measures the input image reconstruction according to 4 It is the difference between the observed input image and its reconstruction by the rendering-based formation model:
L, At; B) = || f(.,r Ar, t, At: B)
(5)
We noticed that (5) is directly related to the quality of reconstruction and deblurring as measured by the evaluation metrics PSNR and SSIM on the FMO deblurring benchmark [IT] (see supplementary). However, directly minimizing the loss (5) is challenging since it is under-constrained and with many local minima, observed experimentally. Therefore, additional priors and regularizers vital.
Silhouette consistency loss. Another important prior that the rendered silhouettes should stay close to sub-frame masks M, estimated by DeFMO []]. This guides the 3D shape estimation

---

0.826
31.98
11
0.849
33.07
aerobie
SfR
GT
0.767
25.29
11
0.908
28.11
volleyball
SYB
GT
0.663
27.21
11
SSIM 0.823
PSNR 29.41
football
S98
G1
1ii11
Temporal super-resolution
Figure 3: Results on the TbD-3D (3 dataset. We compare the proposed Shape- from-Blur (SfB) method with the previous state-of-the-ar DeFMO [I]] and also show the ground truth from high-speed camera (GT). The actual input image (GT row) almost indistinguishable from the image i (II) and SIB row) rendered as a mixture the background and the reconstructed object appearance, temporally averaged over the image exposure time (shown above the background).
with the approximate object location the image, which helps especially in the early steps of the optimization process. The silhouette consistency is defined
Cs(O,r, Ar, t, At) = -1 LoU(MM, M,, +T- (t)+)))tt))- dr
(6)
where the Intersection over Union (IoU) is computed between the mesh silhouettes estimated by our method and the input DeFMO masks.
Laplacian shape regularization loss. Due to the 3D-to-2D projection, many meshes generate the same input image, especially in the mesh areas not visible in the image ('back sides' of the object).
Thus, in order to favor smooth meshes, we add a commonly used 24 26 25 Laplacian shape regularization (e).
Texture smoothness loss. We favor smooth piecewise-constant textures by adding a total variation loss (T(O) on the estimated texture map, which is sparse gradient prior (39], computed as the mean absolute difference between neighboring pixels in both image directions. Thanks to this term, the texture in unobserved mesh parts is smooth transition between the edges of the observed parts. Without (r)) our method produces random noise pixels the unobserved texture parts, which degrade the quality of novel view synthesis.
Joint loss. The joint loss is a weighed sum of the image formation loss 5). the silhouette consis- tency 66. Laplacian shape regularization, and texture smoothness:
C C, Ar, At; Ls ( Ar, At) C (r(
5

---

In Sec. 3.2] motion by test-time optimization on the input image I and background B.
we infer the parameters (θ,r, Ar, t, At) describing the object texture, shape, and
Technical details. The weight of the Laplacian shape regularization is set to Xz. = 1000. There are no weights for other terms since the default unit weight produced good results. We use the Kaolin [40]
implementation of DIB-R [24], and set the smoothness term for soft mask rasterization to 7000- We use a median of the previous five frames in a video as an approximation of the background.
3.2 Loss optimization
We optimize the joint loss 7 using ADAM 41 with the fixed learning rate 0.1 for 500 iterations in the PyTorch (42 framework. This is made possible by using a differentiable renderer, which enables to compute gradients of the rendered image w.r.t. the object parameters. Optimization takes about 60 seconds on a single 8 GB Nvidia GTX 1080 GPU (for 500 iterations of single prototype mesh with 1212 vertices, 2420 faces, 100 100 texture map, 8 time steps to discretize the intergral, and input image of 240 x 320 pixels).
Prototypes. We provide the method with set of three prototypes. The best prototype is automati- cally selected based on the lowest value of the image formation loss (5) after optimization starting from each prototype in turn. The first prototype is sphere with 1212 vertices, 2420 faces, and simple spherical projection texture mapping- The second prototype is bigger sphere with 1538 vertices, 3072 faces, and with more sophisticated Voronoi texture mapping (43. The third prototype is torus with a similar number of vertices and faces and the same Voronoi mapping. We used two spheres with different number of vertices to include more diversity in the complexity of the genus 0 shape. More vertices allow modeling more local shape details, e.g. concavities.
Normalization and initialization. The mesh vertices are normalized to zero center of mass and unit variance, maintaining the normalization throughout the whole optimization process. Normalization to unit variance is needed to keep the mesh in the canonical space. Without such normalization, it would be ambiguous whether the mesh vertices should be moved or the initial pose {r, t} changed. Moreover, this normalization makes it easier to keep the object projection within the image bounds.
Texture values normalized by Sigmoid activation. Translation vectors and t + At estimated as a hyperbolic tangent (tanh) activation with normalization such that the 2D object projection is always visible in the image, i.e. (-1,-1) is the bottom left corner of the image, (1, 1) is the top right corner, a coordinate of 1 indicates an object covering the whole field-of-view (FOV), and = = 1 indicates covering 5% of FOV. Rotations r, Ar are represented by quaternions, and Ar is capped to represent at most 120 degrees total rotation. The mesh is initialized with a uniform grey texture, zero vertex offsets, zero rotation, zero translation, and placed in the center of the image. Experimentallyy initializing the texture by projecting the blurry input image directly onto the mesh led to inferior performance. The reason might be that blurry object appearance is local optimum for the texture, making it more difficult for the optimization process to converge to the sharp object appearance.
In sum, the proposed method estimates the object's textured mesh O and its motion {r, Ar, t, At) by test-time optimization. The input image is explained by the object parameters via minimizing the joint loss 7 that consists of the image formation loss (5) and suitable regularizers.
Evaluation
Datasets. We evaluate our method on four datasets. Three real-world datasets from the FMO
deblurring benchmark [T]: (1) TbD [18), with 12 sequences (471 frames total); (2) TbD-3D (23).
with 10 sequences (516 frames); (3) Falling Objects (22). with 6 sequences (94 frames); (4) and a
synthetic dataset from [I]. The easiest dataset is TbD [18] with uniformly colored, mostly spherical objects. TbD-3D 23 is more difficult and contains objects with complex texture and significant 3D rotation, but the objects are still mostly spherical. Falling Objects [22 is the most challenging dataset, with objects of complex 3D shape (e.g. key, pen, marker) and significant rotation. The ground truth for all three real-world datasets includes the high-speed camera footage at 8 higher frame rate, as well as the sub-frame 2D object locations and 2D masks. We use the synthetic dataset to evaluate the 3D shape O, 3D translation At, and 3D rotation Ar estimated by our method, as there are no real-world datasets of blurred objects with 3D models annotations. The dataset is created by applying the rendering-bel based image formation model 4 to synthetically moving 3D ShapeNet 44

---

Table 1: Evaluation on FMO benchmark. The best- persorrero method is highlighted. Generic
deblurring methods 2 A
do not estimate the object trajectory, thus their TloU is not defined (N A).
The datasets are sorted from the most challenging one [22] to the easiest one [8.
Falling Objects
22
TbD-3D Dataset
23
TbD Datasel
18
Time [s]
Method
TloUt
PSNR*
SSIM
TloUt
PSNRT
SSIM
TloUt
PSNR
SSIM
et al.
N/
23.54
0.575
N/A A
24.52
0.590
N/A
24.90
0.530
DeblurGAN
2
N/A
23.36
0.588
N/A
23.58
0.603
N/A
24.27
0.537
0.1
TbD [8]
0.539
20.53
0,591
0.598
18.84
0.504
0,542
23.22
0.605
100
TbD-3D 23
0.539
23.42
0.671
0.598
23.13
0.651
0.542
25.21
0.674
1000
DeFMO
0.684
26.83
0.753
0.879
26.23
0.699
0.550
25.57
0.602
0.05
SfB (ours)
0.701
27.18
0.760
0.921
26.54
0.722
0.610
25.66
0.659
1969
objects. The backgrounds are randomly sampled from sequences in the VOT dataset [45]. Rotation and translation are generated randomly, with an average translation of 3x the object size and an average rotation of 100%. This way, we generate 10 images, one for each of 10 different objects.
Evaluation metries. We evaluate the deblurring accuracy by comparing the sub-frame object reconstruction to high-speed camera frames. We use Trajectory-IoU (TloU) [8]. which averages the standard IoU between the estimated silhouette and the ground truth mask along the ground truth trajectory. For evaluating sub-frame appearance, we generate a temporal super-resolution image sequence by applying the rendering- image formation model (4) at a series of time sub-intervals (changing the integration bounds) using the object parameters {θ,r, Ar, t, At) estimated by method. We then compare this generated sequence to the ground truth high-speed frames by the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) metrics.
Next, we evaluate the error in estimating the 3D translation At and 3D rotation Ar that the object undergoes during the camera exposure time. The translation error is measured as cat || At 22, and the rotation error CAr is the angle between Ar and Ar (GT denotes ground truth values). Note that the canonical mesh representation of ShapeNet objects is different from the one estimated by our method, preventing us from evaluating the estimate of the initial 6-DoF pose {r, t).
To measure the 3D shape reconstruction accuracy, we align the estimated @ and ground truth @GT meshes and compute the average bidirectional distance between them as ég = 1/2(D(O, θCT) D(eCT, Q)), where D Q2) is the average distance of each vertex of O to the closest face of 2 The meshes are rescaled to tightly fit a unit cube, thus unit of eg and et is one object size (diameter).
4.1 Comparison to previous methods on TbD, TbD-3D, and Falling Objects datasets
We compare to two generic deblurring methods, DeblurGAN-v2 [ and Jin et al. 3 compare to methods specialized for FMO deblurring, which are the TbD method [8]
We also
TbD-3D
method 23 and the -fate-of-the-as art DeFMO [I.
Our SIB method outperforms the above methods on all metrics on the two most challenging datasets Falling Objects and TbD-3D, setting the new state-of-the-art (T. On the easier TbD dataset, SfB achieves the highest TloU and PSNR, but marginally lower SSIM than TbD-3D, which is specifically designed for uniformly colored spherical objects. We show qualitative results in Fig.
and Fig.
4.2 Ablation study Falling Objects and synthetic datasets
The first block in the ablation study (Table shows that omitting the silhouette consistency loss Cs (6) degrades performance by a wide margin, highlighting its importance. We also experiment with an alternative form of the silhouette consistency loss, which uses simple background subtraction instead of the DeFMO masks within (6). More precisely, we replace M, with the difference D between the input image and the background, thresholded at 0.1 (details in supplementary). Using LD is better than completely omitting any silhouette consistency, but performs worse than Ls. since DeFMO localizes the object with the sub-frame precision, thereby providing strong signal to our method. We also tried another loss Lr that favors the textures rendered by R to be similar (in terms of L2 loss) to the aub-frame appearances F, estimated by DeFMO. As shown in Table 2 this loss
7

---

Table 2: Ablation study. We evaluate variants of the silhouette consistency loss ( the influence of the number of prototypes, of using the Laplacian and texture smoothness terms, and of the number of time steps used for discretizing integrals in ④ The best setting uses all proposed loss terms, all prototypes, and discretization steps, where is the number of ground truth high-speed frames.
Falling Objects dataset
22
Synthetic 3D dataset |
Version
C1 4
TloUt
PSNRT
SSIM
Eat Ear
Time [s]
Silvouetees
SfB (no Ls)
0.0371
0.341
17.94
0.594
2.372
36.0
0.039
58
SIB (CD)
0.0264
0.591
20.71
0.680
1.608
23.9
0.030
58
SIB (Cs and LF)
0.0302
0.674
25.64
0.734
0.553
20,2
0,024
SiB (Cs)
0.0234
0.693
26.30
0.745
0.305
17.1
0.023
63
Temms
SfB (no LL)
0.0269
0.676
25.84
0.735
0.462
14,2"
0.023
SfB (no Lr)
0.0276
0.690
24.98
0.751
0.286
17.5
0.023
62
SfB (all terms)
0.0234
0.693
26.30
0.745
0.305
17.1
0.023
63
SIB (torus)
0,0309
0.677
25,59
0.736
1.741
21.39
0.032
63
Prototypes
SfB (sphere, no rot.)
0.0298
0.691
25.84
0.737
0.835
20.4
0.024
61
SfB (sphere)
0.0234
0.693
26.30
0.745
0.305
17.1
0.023
SfB (2 prototypes)
0.0220
0.695
26.43
0,744
0.304
13.2
0.023
124
SIB (3 prototypes)
0.0219
0.701
26.57
0.748
0.194
11.1
0.023
241
SIB (+ 4 distractors)
0.0215
0.696
26.65
0.748
0.250
15.7
0.022
421
Time steps
SfB (1 x steps)
0,0234
0,693
26.30
0.745
0.305
17.1
0.023
63
StB (3 x steps)
0,0212
0.690
26.36
0,738
0.320
19.5
0.021
158
SIB (5 x steps)
0.0209
0.698
26.39
0.736
0.293
17.0"
0.023
253
SIB (10 8 steps)
0.0210
0.698
26.25
0.734
0.329
18.1
0.022
502
SiB (best of all)
0.0191
0.701
27.18
0.760
0.194
11.0
0.024
1969
degrades the performance since DeFMO appearances are still blurry, noisy, and the set of F, does not correspond to consistent 3D object (Fig. 3).
The second block Table shows the improvements of the Laplacian shape regularization and texture smoothness losses on the Falling Objects dataset. However, since the shapes and textures in the synthetic dataset are significantly more complex (e.g. guitar, jar, skateboard, sofa, bottle, pillow), these losses increase the 3D errors by small margin.
The third block demonstrates the benefits of using more prototypes. In order test the robustness of the automatic prototype selection, we also include additional distractor prototypes (teapot, teddy, bunny, cow) that are similar other objects in the datasets. Adding them changes the performance only marginally since they are rarely selected, demonstrating robustness. Besides, the first two rows of this block also show that ignoring the rotation leads drop in performance.
The last block evaluates step count used for discretizing the integral the rendering- bbsedd image formation model (4). The performance increases with the number of steps, until it saturates at around 5x more steps than the number of ground truth high-speed frames available for each image (i.e. 8).
According to the ablation study on the Falling Objects dataset (Table the best performing version of SiB using all loss terms (including DeFMO masks Cs). 7 prototypes (including distractors), and 5x more integration steps than the desired temporal super-resolution output. This version takes 30 minutes optimization time due to the high number of prototypes and time steps, which improve performance only marginally. While we can run our method in configuration taking only a few minutes, optimization time can still be seen as a limitation. We believe that faster runtime could be achieved by avoiding unnecessary computations and iterations, and leave this as future work.
4.3 Applications to super-resolution and novel view synthesis
The main application of SfB is temporal super-resolution. For videos of motion-blurred objects, SfB can generate a video of arbitrary higher frame rates by correctly modeling the object's 3D shape and motion (Fig. 3 The modeling of textured 3D shape allows to generate novel views of the fast
moving object (Fig. 6). SfB correctly recovers the 3D shape and deblurred texture of various moving objects captured by mobile device (Fig. boots, tape roll) or found on YouTube (Fig. 4 tennis ball).
8

---

11
11
SIB
box
SfB
pen
GT
M
S1B
SIB
11
GT
key
SIB
11
SIB
SBB
III
SIB
GT
GT
iri11
Temporal supor-resolution
11i11 Temporal super-resolution
FMOs in the wild
Figure 4: Qualitative results for temporal super-resolution. Left and middle: Falling Objects
dataset [22]. Right: FMOs in the wild [1I] featuring thrown objects captured by mobile device
camera or objects found on YouTube.
Temporal super-resolution
Figure 5: Influence of image noise and compression artifacts. Input images of blurred flying bird are sorted from the least noisy (top) to the most noisy (bottom). Since compression and image noise are not modelled, they slightly deteriorate the reconstruction quality.
3D object rotation estimation has applications sports analysis, e.g. in professional tennis games. For known intrinsic camera parameters, velocity and rotation can be estimated in real-world units.
4.4 Limitations
Since assume a static camera, the method is not designed deal with blur induced by moving or shaking camera. Experimentally, we observed that the method robust to small amounts of moving camera blur. Among other limitations are long optimization time, rigid body assumptions, and inconsistent estimation of unobserved object parts.
The proposed image formation model does not model any type of image noise, e.g. additive image noise, JPEG artifacts, compression, and others. In general, these factors can deteriorate the perfor- mance of the method (Fig. 5). However, the minimization of the reprojection error suppresses noise, and even the basic image formation model without the image noise still leads satisfactory results in many world scenarios, shown in the experimental section.

---

volleyball
aerobie
key
pen
boot
Inputs
TSR
60
30
20
10
0
+10
+-20
+-30%
+-60
Geometry
Figure 6: Novel view synthesis. From a single blurry image (Ist column), our method generates novel views of the objects in the range -60 to +600 around the input view. We also show temporal super-resolution (TSR) at T E {0, 1] and the reconstructed 3D shape without texture (Geometry).
5 Conclusions and future work
We introduced and tackled novel task estimate the textured 3D shape and motion of fast moving objects given a single image with a known background. The proposed method is the first solution to this problem, and also sets new state of the art on 2D fast moving object deblurring. Since the shape estimator deforms the (often spherical) prototype shape, the shape often remains unchanged along unobserved directions (Fig. 6 pen). To address some of the limitations, future work includes adding shape priors, such as symmetry or data-driven priors, and more powerful shape representations for expressing arbitrary shape topologies. The differentiable SIB loss can be incorporated into deep learning pipelines to benefit from large training datasets. SIB can be generalized to optimize over multiple video frames, rather than just one. For instance, when optimizing the textured 3D shape for several consecutive frames, the object becomes visible from more viewpoints providing more constraints and fewer ambiguities in unobserved parts.
Acknowledgments and Disclosure of Funding
This work was supported by a Google Focused Research Award, Innosuisse grant No. 34475.1 IP-ICT, and research grant by the International Federation of Association Football (FIFA).
References
[1] Y.-Y. Chuang, Curless, Salesin, and Szeliski, "A bayesian approach digital matting." in CVPR, vol. 2. pp. II-II, 2001.
10

---

[2] O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang, "Deblurgan-v2: Deblurring (orders-of-maginn) of-magnitude) faster and better," in ICCV, Oct 2019.
[3] M. Jin, G. Meishvili, and P. Favaro, "Learning extract video sequence from single motion-bburree image," in CVPR, June 2018.
[4] L. Pan, R. Hartley, M. Liu, and Y. Dai, "Phase-only image based kernel estimation for single image blind deblurring," in CVPR, June 2019.
[5] J. Pan, H. Bai, and J. Tang, "Cascaded deep video deblurring using temporal sharpness prior,"
in CVPR, June 2020.
[6] Z. Zhong, Y. Gao, Y. Zheng, and B. Zheng, "Efficient spatio-tempores recurrent neural network for video deblurring," in ECCV, pp. 191-207, Springer, 2020.
[7] Shen, W. Bao, G. Zhai, I.. Chen, X. Min, and 7. Gao, "Blurry video frame interpolation," in CVPR, June 2020.
[8] Zhou, J. Zhang, Pan, H. Xie, W. Zuo, and Ren, "Spatio-temporal filter adaptive network for video deblurring," in ICCV, 2019.
[9] M. Jin, Z. Hu, and P. Favaro, "Learning to extract flawless slow motion from blurry videos," in CVPR, June 2019.
[10] D. Rozumnyi, J. Kotera, F. Sroubek, L. Novotný, and J. Matas, "The world of fast moving objects," in CVPR, pp. 4838-4846, July 2017.
[11] D. Rozumnyi, M. R. Oswald, V. Ferrari, J. Matas, and M. Pollefeys, "Defmo: Deblurring and shape recovery of fast moving objects," in CVPR, (Nashville, Tennessee, USA), Jun 2021.
[12] O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, "Deblurgan: Blind motion deblurring using conditional adversarial networks," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[13] C. Seibold, A. Hilsmann, and P. Eisert, "Model-based motion blur estimation for the improve- ment of motion tracking," " CVIU, vol. 160, pp. 45-56, 2017.
[14] D. Gong, J. Yang. L. Liu, Y. Zhang, Reid, C. Shen, A. Van Den Hengel, and Q. Shi, "From motion blur to motion flow: A deep learning solution for removing heterogeneous motion blur,"
CVPR. pp. 3806-3815, 2017.
[15] Gast, A. Sellent, and S. Roth, "Parametric object motion from blur," in CVPR, pp. 1846-1854 2016.
[16] K. Purohit, A. Shah, and A. Rajagopalan, "Bringin alive blurred moments," in CVPR, pp. 6830- 6839, 2019.
[17] Kotera and F. Šroubek, "Motion estimation and deblurring of fast moving objects," in ICIP, pp. 2860-2864, Oct 2018.
[18] J. Kotera, D. Rozumnyi, F. Šroubek, and J. Matas, "Intra-frame object tracking by deblatting," in ICCVW, Oct 2019.
[19] D. Rozumnyi, Kotera, Šroubek, and J. Matas, "Tracking by deblatting." IJCV, vol. 129, no. 9. 2583-2604, 2021.
[20] D. Rozumnyi, J. Kotera, F. Šroubek, and J. Matas, "Non-causal tracking by deblatting." in GCPR (G. A. Fink, S. Frintrop, and X. Jiang, eds.), (Cham), pp. 122-135, Springer International Publishing, 2019.
[21] Šroubek and Kotera, "Motion blur prior," ICIP, pp. 928-932, 2020.
[22] J. Kotera, J. Matas, and F. Šroubek, "Restoration of fast moving objects," IEEE TIP, vol. 29, pp. 8577-8589, 2020.
[23] D. Rozumnyi, J. Kotera, F. Šroubek, and J. Matas, "Sub-frame" appearance and 6d pose estima- tion of fast moving objects," in CVPR, pp- 6777-6785, 2020.
[24] W. Chen, J. Gao, H. Ling, E. Smith, J. Lehtinen, A. Jacobson, and S. Fidler, "Learning to predict differentiable renderer," " in NeurIPS, 2019.
3d objects with an interpolation
[25] Liu, T. Li, W. Chen, and H. Li, "Soft rasterizer: A differentiable renderer for image-based 3d reasoning," ICCV, Oct 2019.

---

[26] N. Wang. Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang, "Pixel2mesh: Generating 3d mesh models from single rgb images." in ECCV, 2018.
[27] S. Popov, P. Bauszat, and V. Ferrari, "Corenet: Coherent 3d scene reconstruction from single rgb image." in ECCV (A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, eds.), (Cham), pp. 366-383, Springer International Publishing, 2020.
[28] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis," in ECCV, 2020.
[29] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, "pixelnerf: Neural radiance fields from one or few images," 2020.
[30] K. Rematas, R. Martin-Brualla, and V. Ferrari, "Sharf: Shape-conditioned radiance fields from single view," CoRR, vol. abs/2102.08866 2021.
[31] E. Prados and O. Faugeras, Shape From Shading, pp. 375-388. Boston, MA: Springer US, 2006.
[32] K. ichi Kanatani and T.-C. Chou, "Shape from texture: General principle," Artificial Intelligence, vol. 38, no. 1, pp. 1-48, 1989.
[33] S. Nayar and Y. Nakagawa, "Shape from focus," JEEE TPAMI, vol. 16, no. 8, pp. 824-831, 1994.
[34] P. Favaro and S. Soatto, "A geometric approach shape from defocus." IEEE TPAMI, vol. 27, no. 3, pp. 406-417, 2005.
[35] S. Savarese, M. Andreetto, H. Rushmeier, F. Bernardini, and P. Perona, "3d reconstruction by shadow carving: Theory and practical evaluation," ICV, vol. 71, no. 3, 2007.
[36] Y. Adato, Y. Vasilyev, O. Ben-Shahar, and T. Zickler, "Toward a theory of shape from specular flow," in ICCV, pp. 1-8, 2007.
[37] W. Skarbek, "Shape from motion revisited," in Active Media Technology (D. Šlezak, G. Schaefer, S. T. Vuong, and Y.-S. Kim, eds.), (Cham), pp. 383-394, Springer International Publishing, 2014.
[38] S. N. Sinha and M. Pollefeys, "Multi-view reconstruction using photo-consistency and exact silhouette constraints: A maximum-flow formulation," in ICCV, ICCV '05, (USA), p. 349-356, IEEE Computer Society, 2005.
[39] D. Perrone and P. Favaro, "Total variation blind deconvolution: The devil is in the details," in CVPR, pp. 2909-2916, 2014.
[40] K. M. Jatavallabhula, E. Smith, J.-F. Lafleche, C. F. Tsang, A. Rozantsev, W. Chen, T. Xiang, R. Lebaredian, and S. Fidler, "Kaolin: A pytorch library for accelerating 3d deep learning research," arXiv:1911.0504 2019.
[41] D. Kingma and Ba, "Adam: A method for stochastic optimization," in ICLR (Y. Bengio and Y. LeCun, eds.), 2015.
[42] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf. E. Yang. Z. Vito,, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library," in Advances in Neural Information Processing Systems 32 (H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Gamett, eds.), Curran Associates, Inc., 2019.
pp.
[43] U. Pinkall and K. Polthier, "Computing discrete minimal surfaces and their conjugates." Experi- mental Mathematics, vol. 2, no. 1, pp. 15 - 36, 1993.
[44] A. X. Chang, Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, "ShapeNet: An Information-Rich 3D Model Repos- itory," Tech. Rep. 1rXiv:1512.0301 [cs. GRJ, Stanford University Princeton University Toyota Technological Institute Chicago, 2015.
[45] M. Kristan, J. Matas, A. Leonardis, T. Vojir, R. Pflugfelder, G. Fernandez, G. Nebehay, F. Porikli, and L. Čehovin, "A novel performance evaluation methodology for single-target trackers," IEEE TPAMI, vol. 38, pp. 2137-2155, Nov 2016.
12


Reviews:
[{"review_id": "r1oxDmseef", "reviewer": "Reviewer_VMLD", "summary": "This paper introduces an approach for predicting the textured 3D shape and motion from blurred video frames. The approach is based on the differentiable rendering that can predict shapes and image formations that can be regarded as a combination of background and foreground scenes. The paper shows interesting results that can reasonably recover shapes, motion, and a deblurred texture map.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces an interesting approach to recover 3D shapes from challenging video frames. The approach is straightforward to understand and easy to follow. The overall pipeline that consists of mask prediction, object prediction, appearance prediction module seems to be adequately designed. The produced output is obviously superior to deblurring approaches as well as a concurrent approach called DeFMO [11].\n\nPros:\n\n + The paper proposed an approach that can jointly estimate the 3D shape, texture, and motion of an input object. The visual quality of the demonstrate results is quite impressive. It seems to be pretty hard to predict the shapes from the images, but the recovered scene is quite reasonable. It is also surprising that it is possible to recover shape and textures from such challenging input images. I am not fully following this area, so my judgment may not be accurate, but it is impressive.\n\n + The experiment is carefully designed to demonstrate the effectiveness of the proposed approach. The authors use the multiple evaluation metrics (L_I, TIoU, PSNR, SSIM) to evaluate the natural scenes and use translation or rotational error for the synthetic dataset. Given the constraint that cannot capture the ground truth of the shape (except for the synthetic case), the demonstration is reasonable. In addition, authors design self-baselines (such as L_d and L_f) to demonstrate the effectiveness of the proposed loss functions.\n\n+ Authors provide the source code of the proposed approach. There was not enough time to go over the detailed validation of the submitted code as an emergency reviewer. However, such a submission could help to understand the proposed approach.\n\nCons:\n\n - It is not fully convincing the importance of the problem. In 26-29, “Our work could eventually make FMO tracking accessible to everyone using regular cameras rather than expensive high-speed cameras, or push the capabilities of current high-speed cameras to a new level…” I was expecting some challenging cases after reading the introduction, but the demonstrated image examples are pretty constrained. The camera stands still, and the single object is assumed to have constant rotational and translational velocity. Because of these, the demonstrated results look unrealistic to me. There are examples using the YouTube videos, but it is still not clear these are the natural scenes that “every people” (as stated in the introduction) may encounter. Rather than that, the YouTube examples seem to be carefully chosen by the authors. It is because the YouTube videos are pretty similar to the Falling Objects [21] and TbD-3D dataset [22], and TbD dataset [18]. The authors mention this issue as the limitation in Sec 5, but it does not entirely resolve the concern. How about just adding failure cases and include thorough discussions regarding them? This could bring a better understanding of the proposed approach.\n\n- The proposed approach is a collection of well-known modules. The authors use DeFMO for the silhouette consistency prediction and differentiable interpolation-based render (DIB-R) [23] and Kaolin [39] for the image formation check. The approach combines these modules to solve the problem differently. During this procedure, there is no new module is proposed.\n\n - It is hard to understand the computational complexity of the proposed approach. Can authors compare the computation time with other approaches? Table 2 shows the time for the proposed approach with different configurations, but it does not tell how it is faster or slower than the baseline approaches. How about comparing the runtime with other approaches such as Jin et al. [3], DeblurGAN [2], TbD [18], TbD-3D [22], and DeFMO [11]?\n\n- A sentence shown in lines 169-170 is quite misleading. It states, “input image of 240x320 pixels takes around *60* seconds…” This is a misleading statement since the best performing approach takes *1969* seconds, as shown in Table 2. The performance of the configuration that takes 30 mins is reported in Table 1, and the line 169-170 below Table 1 says it takes about a minute. This is how the paper can give the wrong message to readers. Instead, the configurations that take about 60 seconds are comparable to DeFMO [11], based on TIoU, PSNR, and SSIM as shown in Table 1 and Table 2. I suspect the differentiable rendering takes longer if the number of prototypes and the proposed loss terms are applied. Long optimization time could be the limitation of the proposed approach, and it should be properly mentioned in the paper.\n \n- Regarding the three prototypes, it is not clear why the approach should have a small sphere and the bigger sphere. For readers, ‘big’ and ‘small’ is quite ambiguous since it does not provide any detailed information regarding the size of spheres. If the approach should have two kinds of spheres, why not using two different sizes of ‘torus’? This question and ambiguity make it hard to understand the intention of the prototypes.\n\n- Is there any consideration on the additive image noise? The basic image formulation seems to ignore the effect of additive noise on the image. However, the target problem is to tackle challenging images and recover useful information, as highlighted in the introduction of the paper. Therefore, it is necessary to add a discussion of how the proposed approach could or could not handle challenging cases having noises. Such examples could be included in the limitation of the proposed approach (see my previous proposal regarding the limitations), and it could provide more information when the proposed approach works as expected or not.\n\nOverall, the paper has several concerns, but this paper shows clear merits – the approach demonstrates the new pipeline can handle challenging blurred images of moving objects. I am leaning toward accepting for now. However, I am not an expert in this area, and I am not sure the NeurIPS venue would be the best fit for this problem. The confidence is not relatively high.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to the questions in the main review.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hrs"}, "extra_text": {}}, {"review_id": "RObEND-sAr", "reviewer": "Reviewer_qzSX", "summary": "The submitted paper focuses on the shape, texture, and motion recovery of a single fast moving object from two input images from a static camera - a clean background image, and a blurred image due to fast object motions. The authors tackle the problem using differential rendering and iterative optimization at test time. Specifically, the template mesh (\"prototype\"), the texture map, the initial translation and rotation, as well as the constant motion speed are jointly optimized in order to minimize a combination of loss functions, with the image formation loss being the key contribution. The method is demonstrated on both synthetic dataset and relatively simple real datasets. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n\nThe proposed method is an application of differential rendering in the context of deblurring - several images are rendered separately and then integrated to form a blurred image, and then loss is back-propagated to rectify the shape, texture, and motion model. To the best of my knowledge, this proposed method differs from most of the recent SotA methods for deblurring, which focus on supervised learning and training on a large dataset.\n\n\nQuality \n\nThe proposed method exceeds several prior work on the FMO benchmark. Based on the videos and images shown, the reconstructed mesh is reasonably close to the ground truth (considering that this is a heavily under-constrained problem, perfect reconstruction isn't possible). However, I suspect that the improvement in accuracy over prior methods is largely attributed to the fact that the template mesh is quite close the the ground truth in shape. This is verified by the prototypes study in Table 2 on Synthetic 3D dataset - when the ground truth shapes become slightly more complicated, the performance gap is significant (0.305 vs. 0.194) when different number of prototypes is used. Additionally, it is unclear how import some of the optimized variables are (e.g., rotational speed, texture optimization) and how much they contribute to the final accuracy\n\nI think a few ablation studies can greatly improve the quality of the paper. \n1. an ablation study on different mesh topologies as prototypes (when only 1 prototype is used), ideally we see how the performance drops when the topology deviates from the ground truth (e.g., torus vs. sphere)\n2. an ablation study on the contribution of rotational speed\n3. an ablation study on the contribution of texture optimization, if the texture is initialized properly (e.g., by projecting the image pixels directly onto the mesh)\n4. how does the method perform on \"slow-motion\" blur, especially compared to prior work? \n\n\nClarity\n\nThe paper is overall well written and easy to follow, except for a few things:\n1. Why is normalizing to unit variance needed? This is in the \"Normalization and initialization\" paragraph under Section 3.2\n2. How is the number of subframes and the time difference selected?\n\n\nSignificance\n\nThe proposed method is a proof-of-concept on how differential rendering and a 3D prior can help improve 2D deblurring. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "9-7MfMACkmq", "reviewer": "Reviewer_52ha", "summary": "This paper presents SfB, a novel method of handling a single RGB image of a significantly blurred fast moving object, by recovering the object's shape, texture, and trajectory, what are optimized towards a joint loss of image consistencies, via a differentiable render. The method can be applied to a variant of tasks of de-blurring, shape reconstruction, super resolution and view synthesis. The paper reports state-of-the-art results on a variant of real-world and synthetic datasets, both quantitatively and qualitatively.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While relying on some of the previous work's output, this paper presents a novel method that has not seen in related works. The formulation of this method is intuitive and clear, and the discussions are comprehensive.\nThe proposed method is robust against real-world inputs, and achieves state-of-the-arts results.\nThis paper is very well written, easy to follow, and contains all the details for re-implementing. The author also provided their code in the supplement materials. \nThough the reviewer believes the most weakness of this paper comes from its shape representation(see Limitations), it does not void the core idea of retrieving 3D from blurred images, and further study is encouraged along the track of improving 3D reconstructions.\nOverall, it is a strong paper that would be a good contribution to the community.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Shape representation. \na) As the method relies on the limited set of fixed topology prototypes, it is hard to generalize to arbitrary or highly complicated shapes, as also stated in the paper. This is also a constraint from the used render, which requires such mesh format. Can the shape representation be improved to overcome this limitation(such as, can NeRF-like shape and render mechanism be used here?) Or can the prototype can reconstructed in the first place?\nb) Rigid assumption. During the exposure time, the object is assumed to be of fixed shape, only changing its position by rigid transformation. It is not mentioned in the paper whether \\Theta is allowed to change during the exposure time. Can this method be applied to non-rigid objects, such as clothes or moving parts? (For instance, can \\Theta mesh representation dependent on time \\t?)\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "29yzIkVBRb4", "reviewer": "Reviewer_QNgV", "summary": "This paper presents a method for 3D shape reconstruction of objects with large motion blurs in a video sequence. The authors propose to augment an off-the-shelf 2D shape-from-blur method, whose output is used to fit a 3D mesh shape as well as a linear 3D motion trajectory within a small timeframe. The motion blur is synthesized by averaging the shape & texture predictions within the timeframe for the image reconstruction loss, in addition to other regularization terms to keep the optimization well-behaved. Temporal super-resolution can also be achieved by synthesizing the image at a continuous timestamp. Experiments are benchmarked on 3 real-world datasets containing sequences of small objects and a synthetic dataset for benchmarking reconstruction accuracy.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper is very nicely presented. The introduction and related work sections give a good overview of the importance of the problem of shape estimation from blurred images and clearly explain where this paper stands in the problem scope. \n+ The paper proposes a simple and neat way of combining 2D shape-from-blur methods with differentiable mesh renderers to further predict the 3D shape structures. The problem formulation and method descriptions are also well-explained.\n+ The 3D shape predictions are successful in most cases. The experiments show a clear advantage of how 3D prediction and modeling explicit 3D transformations help synthesize higher quality images for temporal super-resolution as well. Sufficient ablation studies are also provided for understanding the contribution from each component.\n\nWeaknesses:\n- I think it would be much more helpful if the method could be evaluated on datasets with larger objects and of higher resolutions. I'm not sure if this was intended to target low-resolutional images so that motion blurs are more significant, but the objects evaluated with seem to be of very low resolution, and it is difficult to visually compare the results (e.g. the temporal super-resolution results in Fig. 4 are not very informative, as everything is rather still blurry). I personally think only the volleyball, tape, and shoe examples are more discriminative.\t\n- The explanation of prototypes is not very clear. Do the authors mean different shape templates with different genus numbers? It is not clear why spheres of different sizes (L171-176) are considered two different prototypes (shape templates), as I would expect the silhouette loss to find the right scale automatically. Also, it is not clear what \"distractor prototypes\" are (L247). My understanding is that a torus and a teapot would both be genus-1 and the rest genus-0.\n- As all the objects considered in the paper are either genus-0 or 1, it would be helpful to discuss how the proposed framework (including DeFMO) could be extended to arbitrary shape topologies.\n- As I understand, DeFMO also predicts the image appearance in addition to the silhouettes. Why isn't the predicted appearance utilized instead of the proposed mesh texture prediction? I would think it should make the optimization problem much more constrained as well.\n\nAlso, as the 3D transformations considered are linear with time (within the small timeframe), I think it would be helpful if the authors could provide some thoughts on whether such 3D motion blur could be more analytically synthesized without having to average multiple rendered samples across time, so as to more faithfully approximate the actual integral.\n\nOther minor problems:\n- The paragraphs in L242-245 and L246-249 refers to the wrong blocks and should be swapped.\n- What does $\\hat{I}/I$ in Fig. 3, 4 stand for? It's not clear which images correspond to $\\hat{I}$ and $I$ respectively.\n- The claim in L103 of estimating 3D shape from a single image sounds invalid and contradicts with L106 (background is also assumed given).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6hr"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main task of this work is fast motion deblurring and yet the paper claims to jointly estimate 3D shape and texture along with the motion of the object. The paper received positive reviews: 7, 9, 6, 6. The paper is written well with impressive results compared to previous works. However, the main limitations of the paper is the lack of novelty. As pointed out by a reviewer, the proposed approach is  an extension of the DeFMO paper. The authors use DeFMO for the silhouette consistency prediction and differentiable interpolation-based render (DIB-R) and Kaolin [39] for the image formation check. The approach combines these modules to solve the problem differently. While the DeFMO directly renders the appearance and silhouette directly from the given input image and estimated background, this work instead estimates a 3D shape from the given input image (and background) and then renders the appearance and silhouette.
It is also worthy to note that estimating 3D shape and differentiable rendering from an image have been previously studied [23, 24, 25, 26]. The fact that this work estimates it from a motion-blur image doesn't necessarily make it significantly different from previous works. Moreover, the 3D estimation from the motion-blur image is also constrained with specific types of prototypes which makes it less generalizable to all types of fast-moving objects.

The approach demonstrates a new pipeline for jointly estimating shape, texture, and motion with challenging blurred images of moving objects. I am not sure the NeurIPS venue would be the best fit for this problem. Based on the reviews and the author rebuttals, I lean toward to reject the paper.

Decision: Accept (Poster)