Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces an approach for predicting the textured 3D shape and motion from blurred video frames. The approach is based on the differentiable rendering that can predict shapes and image formations that can be regarded as a combination of background and foreground scenes. The paper shows interesting results that can reasonably recover shapes, motion, and a deblurred texture map.
main_review: The paper introduces an interesting approach to recover 3D shapes from challenging video frames. The approach is straightforward to understand and easy to follow. The overall pipeline that consists of mask prediction, object prediction, appearance prediction module seems to be adequately designed. The produced output is obviously superior to deblurring approaches as well as a concurrent approach called DeFMO [11].

Pros:

 + The paper proposed an approach that can jointly estimate the 3D shape, texture, and motion of an input object. The visual quality of the demonstrate results is quite impressive. It seems to be pretty hard to predict the shapes from the images, but the recovered scene is quite reasonable. It is also surprising that it is possible to recover shape and textures from such challenging input images. I am not fully following this area, so my judgment may not be accurate, but it is impressive.

 + The experiment is carefully designed to demonstrate the effectiveness of the proposed approach. The authors use the multiple evaluation metrics (L_I, TIoU, PSNR, SSIM) to evaluate the natural scenes and use translation or rotational error for the synthetic dataset. Given the constraint that cannot capture the ground truth of the shape (except for the synthetic case), the demonstration is reasonable. In addition, authors design self-baselines (such as L_d and L_f) to demonstrate the effectiveness of the proposed loss functions.

+ Authors provide the source code of the proposed approach. There was not enough time to go over the detailed validation of the submitted code as an emergency reviewer. However, such a submission could help to understand the proposed approach.

Cons:

 - It is not fully convincing the importance of the problem. In 26-29, “Our work could eventually make FMO tracking accessible to everyone using regular cameras rather than expensive high-speed cameras, or push the capabilities of current high-speed cameras to a new level…” I was expecting some challenging cases after reading the introduction, but the demonstrated image examples are pretty constrained. The camera stands still, and the single object is assumed to have constant rotational and translational velocity. Because of these, the demonstrated results look unrealistic to me. There are examples using the YouTube videos, but it is still not clear these are the natural scenes that “every people” (as stated in the introduction) may encounter. Rather than that, the YouTube examples seem to be carefully chosen by the authors. It is because the YouTube videos are pretty similar to the Falling Objects [21] and TbD-3D dataset [22], and TbD dataset [18]. The authors mention this issue as the limitation in Sec 5, but it does not entirely resolve the concern. How about just adding failure cases and include thorough discussions regarding them? This could bring a better understanding of the proposed approach.

- The proposed approach is a collection of well-known modules. The authors use DeFMO for the silhouette consistency prediction and differentiable interpolation-based render (DIB-R) [23] and Kaolin [39] for the image formation check. The approach combines these modules to solve the problem differently. During this procedure, there is no new module is proposed.

 - It is hard to understand the computational complexity of the proposed approach. Can authors compare the computation time with other approaches? Table 2 shows the time for the proposed approach with different configurations, but it does not tell how it is faster or slower than the baseline approaches. How about comparing the runtime with other approaches such as Jin et al. [3], DeblurGAN [2], TbD [18], TbD-3D [22], and DeFMO [11]?

- A sentence shown in lines 169-170 is quite misleading. It states, “input image of 240x320 pixels takes around *60* seconds…” This is a misleading statement since the best performing approach takes *1969* seconds, as shown in Table 2. The performance of the configuration that takes 30 mins is reported in Table 1, and the line 169-170 below Table 1 says it takes about a minute. This is how the paper can give the wrong message to readers. Instead, the configurations that take about 60 seconds are comparable to DeFMO [11], based on TIoU, PSNR, and SSIM as shown in Table 1 and Table 2. I suspect the differentiable rendering takes longer if the number of prototypes and the proposed loss terms are applied. Long optimization time could be the limitation of the proposed approach, and it should be properly mentioned in the paper.
 
- Regarding the three prototypes, it is not clear why the approach should have a small sphere and the bigger sphere. For readers, ‘big’ and ‘small’ is quite ambiguous since it does not provide any detailed information regarding the size of spheres. If the approach should have two kinds of spheres, why not using two different sizes of ‘torus’? This question and ambiguity make it hard to understand the intention of the prototypes.

- Is there any consideration on the additive image noise? The basic image formulation seems to ignore the effect of additive noise on the image. However, the target problem is to tackle challenging images and recover useful information, as highlighted in the introduction of the paper. Therefore, it is necessary to add a discussion of how the proposed approach could or could not handle challenging cases having noises. Such examples could be included in the limitation of the proposed approach (see my previous proposal regarding the limitations), and it could provide more information when the proposed approach works as expected or not.

Overall, the paper has several concerns, but this paper shows clear merits – the approach demonstrates the new pipeline can handle challenging blurred images of moving objects. I am leaning toward accepting for now. However, I am not an expert in this area, and I am not sure the NeurIPS venue would be the best fit for this problem. The confidence is not relatively high.
limitations_and_societal_impact: Please refer to the questions in the main review.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper presents SfB, a novel method of handling a single RGB image of a significantly blurred fast moving object, by recovering the object's shape, texture, and trajectory, what are optimized towards a joint loss of image consistencies, via a differentiable render. The method can be applied to a variant of tasks of de-blurring, shape reconstruction, super resolution and view synthesis. The paper reports state-of-the-art results on a variant of real-world and synthetic datasets, both quantitatively and qualitatively.
main_review: While relying on some of the previous work's output, this paper presents a novel method that has not seen in related works. The formulation of this method is intuitive and clear, and the discussions are comprehensive.
The proposed method is robust against real-world inputs, and achieves state-of-the-arts results.
This paper is very well written, easy to follow, and contains all the details for re-implementing. The author also provided their code in the supplement materials. 
Though the reviewer believes the most weakness of this paper comes from its shape representation(see Limitations), it does not void the core idea of retrieving 3D from blurred images, and further study is encouraged along the track of improving 3D reconstructions.
Overall, it is a strong paper that would be a good contribution to the community.

limitations_and_societal_impact: 1. Shape representation. 
a) As the method relies on the limited set of fixed topology prototypes, it is hard to generalize to arbitrary or highly complicated shapes, as also stated in the paper. This is also a constraint from the used render, which requires such mesh format. Can the shape representation be improved to overcome this limitation(such as, can NeRF-like shape and render mechanism be used here?) Or can the prototype can reconstructed in the first place?
b) Rigid assumption. During the exposure time, the object is assumed to be of fixed shape, only changing its position by rigid transformation. It is not mentioned in the paper whether \Theta is allowed to change during the exposure time. Can this method be applied to non-rigid objects, such as clothes or moving parts? (For instance, can \Theta mesh representation dependent on time \t?)
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The submitted paper focuses on the shape, texture, and motion recovery of a single fast moving object from two input images from a static camera - a clean background image, and a blurred image due to fast object motions. The authors tackle the problem using differential rendering and iterative optimization at test time. Specifically, the template mesh ("prototype"), the texture map, the initial translation and rotation, as well as the constant motion speed are jointly optimized in order to minimize a combination of loss functions, with the image formation loss being the key contribution. The method is demonstrated on both synthetic dataset and relatively simple real datasets. 
main_review: Originality

The proposed method is an application of differential rendering in the context of deblurring - several images are rendered separately and then integrated to form a blurred image, and then loss is back-propagated to rectify the shape, texture, and motion model. To the best of my knowledge, this proposed method differs from most of the recent SotA methods for deblurring, which focus on supervised learning and training on a large dataset.


Quality 

The proposed method exceeds several prior work on the FMO benchmark. Based on the videos and images shown, the reconstructed mesh is reasonably close to the ground truth (considering that this is a heavily under-constrained problem, perfect reconstruction isn't possible). However, I suspect that the improvement in accuracy over prior methods is largely attributed to the fact that the template mesh is quite close the the ground truth in shape. This is verified by the prototypes study in Table 2 on Synthetic 3D dataset - when the ground truth shapes become slightly more complicated, the performance gap is significant (0.305 vs. 0.194) when different number of prototypes is used. Additionally, it is unclear how import some of the optimized variables are (e.g., rotational speed, texture optimization) and how much they contribute to the final accuracy

I think a few ablation studies can greatly improve the quality of the paper. 
1. an ablation study on different mesh topologies as prototypes (when only 1 prototype is used), ideally we see how the performance drops when the topology deviates from the ground truth (e.g., torus vs. sphere)
2. an ablation study on the contribution of rotational speed
3. an ablation study on the contribution of texture optimization, if the texture is initialized properly (e.g., by projecting the image pixels directly onto the mesh)
4. how does the method perform on "slow-motion" blur, especially compared to prior work? 


Clarity

The paper is overall well written and easy to follow, except for a few things:
1. Why is normalizing to unit variance needed? This is in the "Normalization and initialization" paragraph under Section 3.2
2. How is the number of subframes and the time difference selected?


Significance

The proposed method is a proof-of-concept on how differential rendering and a 3D prior can help improve 2D deblurring. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method for 3D shape reconstruction of objects with large motion blurs in a video sequence. The authors propose to augment an off-the-shelf 2D shape-from-blur method, whose output is used to fit a 3D mesh shape as well as a linear 3D motion trajectory within a small timeframe. The motion blur is synthesized by averaging the shape & texture predictions within the timeframe for the image reconstruction loss, in addition to other regularization terms to keep the optimization well-behaved. Temporal super-resolution can also be achieved by synthesizing the image at a continuous timestamp. Experiments are benchmarked on 3 real-world datasets containing sequences of small objects and a synthetic dataset for benchmarking reconstruction accuracy.
main_review: Strengths:
+ The paper is very nicely presented. The introduction and related work sections give a good overview of the importance of the problem of shape estimation from blurred images and clearly explain where this paper stands in the problem scope. 
+ The paper proposes a simple and neat way of combining 2D shape-from-blur methods with differentiable mesh renderers to further predict the 3D shape structures. The problem formulation and method descriptions are also well-explained.
+ The 3D shape predictions are successful in most cases. The experiments show a clear advantage of how 3D prediction and modeling explicit 3D transformations help synthesize higher quality images for temporal super-resolution as well. Sufficient ablation studies are also provided for understanding the contribution from each component.

Weaknesses:
- I think it would be much more helpful if the method could be evaluated on datasets with larger objects and of higher resolutions. I'm not sure if this was intended to target low-resolutional images so that motion blurs are more significant, but the objects evaluated with seem to be of very low resolution, and it is difficult to visually compare the results (e.g. the temporal super-resolution results in Fig. 4 are not very informative, as everything is rather still blurry). I personally think only the volleyball, tape, and shoe examples are more discriminative.	
- The explanation of prototypes is not very clear. Do the authors mean different shape templates with different genus numbers? It is not clear why spheres of different sizes (L171-176) are considered two different prototypes (shape templates), as I would expect the silhouette loss to find the right scale automatically. Also, it is not clear what "distractor prototypes" are (L247). My understanding is that a torus and a teapot would both be genus-1 and the rest genus-0.
- As all the objects considered in the paper are either genus-0 or 1, it would be helpful to discuss how the proposed framework (including DeFMO) could be extended to arbitrary shape topologies.
- As I understand, DeFMO also predicts the image appearance in addition to the silhouettes. Why isn't the predicted appearance utilized instead of the proposed mesh texture prediction? I would think it should make the optimization problem much more constrained as well.

Also, as the 3D transformations considered are linear with time (within the small timeframe), I think it would be helpful if the authors could provide some thoughts on whether such 3D motion blur could be more analytically synthesized without having to average multiple rendered samples across time, so as to more faithfully approximate the actual integral.

Other minor problems:
- The paragraphs in L242-245 and L246-249 refers to the wrong blocks and should be swapped.
- What does $\hat{I}/I$ in Fig. 3, 4 stand for? It's not clear which images correspond to $\hat{I}$ and $I$ respectively.
- The claim in L103 of estimating 3D shape from a single image sounds invalid and contradicts with L106 (background is also assumed given).
limitations_and_societal_impact: Adequately addressed
review_title: null

Meta-review: The main task of this work is fast motion deblurring and yet the paper claims to jointly estimate 3D shape and texture along with the motion of the object. The paper received positive reviews: 7, 9, 6, 6. The paper is written well with impressive results compared to previous works. However, the main limitations of the paper is the lack of novelty. As pointed out by a reviewer, the proposed approach is  an extension of the DeFMO paper. The authors use DeFMO for the silhouette consistency prediction and differentiable interpolation-based render (DIB-R) and Kaolin [39] for the image formation check. The approach combines these modules to solve the problem differently. While the DeFMO directly renders the appearance and silhouette directly from the given input image and estimated background, this work instead estimates a 3D shape from the given input image (and background) and then renders the appearance and silhouette.
It is also worthy to note that estimating 3D shape and differentiable rendering from an image have been previously studied [23, 24, 25, 26]. The fact that this work estimates it from a motion-blur image doesn't necessarily make it significantly different from previous works. Moreover, the 3D estimation from the motion-blur image is also constrained with specific types of prototypes which makes it less generalizable to all types of fast-moving objects.

The approach demonstrates a new pipeline for jointly estimating shape, texture, and motion with challenging blurred images of moving objects. I am not sure the NeurIPS venue would be the best fit for this problem. Based on the reviews and the author rebuttals, I lean toward to reject the paper.

Decision: Accept (Poster)