Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new method for reconstructing a 3D shape template and dense 3D part trajectories from a monocular video. This work is very similar to a recent work LASR. The main difference, which is also the main contribution, is to establish long-range correspondences by matching the features between each 2D frame to a 3D canonical shape. 
main_review: This work solves an interesting and important problem, i.e. reconstructing 3D templates and dense trajectories without any 3D coarse geometry or prior knowledge. The idea of establishing correspondences via pixel-to-model registration makes sense, which avoids the drifting issue caused by integrating optical flows frame by frame. 
This method is very close to LASR but achieves better quality. 
My main concern is that this method lacks novelty. Most of the pipeline components are quite similar to LASR. The main difference is how to establish correspondence. Instead of supervising with frame-to-frame optical flows, this method introduces a shared canonical space for employing the pixel-to-model registration. However, this pixel-to-model matching is widely used in classical computer vision and graphics techniques for registration. 
Besides, a comparison with "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes" is expected. 
Also, the authors are expected to release the code and data. 

limitations_and_societal_impact: See above for the limitations. No negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Authors propose a method for reconstructing deforming objects from videos (with the target object pre-segmented). The output is a mesh with bone assignments, plus temporal deformation parameters (i.e. bone transformations). The approach predicts correspondences between frame pixels and points on the mesh by mapping them into a joint embedding space, using a contrastive loss. The method is demonstrated on videos of humans and animals, where it out-performs a recent baseline.
main_review: Method:

++ The proposed approach is novel. More specifically: there is significant commonality with LASR, but I feel there's enough technical novelty in the present work to justify publication.

++ Different parts of the model are clearly motivated and described.

++ The idea of learning a joint embedding space for the 3D surface and the pixels is elegant.

Experiments:

++ The qualitative results on reconstruction are impressive, including on fairly challenging data

++ Quantitative results (though only on keypoint transfer) exceed baselines (which are sensibly-selected modern approaches), apparently by a significant margin

-- There is no discussion of error bars (due e.g. to different initialisations) -- this should be added.

-- There is no direct evaluation of the geometric reconstruction performance. While this is challenging for natural data (lack of gt), it would be valuable to see such results on synthetic data, plus a comparison to baselines.

-- Part-segmentation evaluation is not evaluated numerically -- only qualitative results are shown (which are ok but do not seem particularly accurate). A quantitative evaluation, against a suitable baseline, would be valuable.

-- The predicted textures are never shown.

Misc.:

- Fig.1: typo: "inlcuding"
- 269: typo: "alblations"

---

## Post-rebuttal

The authors' response addresses my concerns sufficiently, and I remain in favor of acceptance having read the other reviews (and respective responses). The sensitivity to initialisation (as reported in the authors' comment below) seems significant, and so this should be discussed in the revised paper. I still think more quantitative evaluation (of geometric reconstruction) would strengthen the paper -- this could be done on synthetic data, or human data with full annotations (e.g. Human3.6M).
limitations_and_societal_impact: There is no discussion of limitations. There is brief (but adequate) discussion of broader societal impacts.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents an optimization-based framework that recovers 3D shape, articulated pose, and weak texture from a monocular video of an articulated object. The proposed method extends a recent method (LASR) with a surface embedding matching module for establishing long-range correspondences, and achieves significantly better results on highly deformable objects, such as break dancers.

This is an extremely challenging task, and has a long history in computer vision. Current methods either require extra information, eg depth or keypoints, or rely on category-specific template shape models. The results achieved by this method is very impressive, relying only on off-the-shelf category-agnostic segmentation and optical flow models.

The paper also demonstrates the possibility of extending the method with different instances in multiple videos, which presents exciting potential of capturing category-level priors for fast inference.

main_review: ## Strengths
### S1 - Method
The proposed extension from LASR is very powerful. It aims at establishing long-range correspondences by matching pixels in different frames through 3D surface embeddings. Specifically, pixels correspondences are established indirectly by matching CNN-predicted pixel embeddings to the same 3D surface embeddings.
At a high level, instead of relying on pixel colors for matching correspondences, it matches correspondences in a learned embedding space, which is more robust against pixel variations resulting from shading, occlusion, deformation etc.
This idea is carefully implemented in this paper and achieves impressive results.


### S2 - Results
- The reconstruction results show significant improvement over the LASR baseline, especially on longer videos, confirming the effectiveness of this embedding correspondence matching module.
- The evaluation and comparison with prior arts are extensive and thorough, presenting lots of technical insights.
- Moreover, the results on multi-video optimization also look very promising, demonstrating the possibility of extracting category-level shape priors from a video collection, which could potentially be extended to unlimited Internet videos at scale.


## Weaknesses
### W1 - Complicated optimization pipeline
One major limitation of the proposed method is its overwhelming complexity. The method is built on top of LASR which already has a very complicated optimization framework, and this new version further adds in even more complexity. It is very impressive that the authors manage to produce those results with this gigantic framework, but it still poses concerns on generalization. However, the paper already shows results on a diversity set of examples, which suggests it's still fairly robust.

### W2 - Texture
Another weakness of the method is the poor texture prediction. I wonder why the optimized texture looks very coarse as shown in the supplementary results.
But as the method mainly focuses on shape reconstruction, and the model bypasses establishing correspondences via pixel colors but rather through embeddings, I think this is not a big issue.


### W3 - Model (more like a question)
Intuitively, I thought the _contrastive matching loss_ $L_\text{ct}$ and the _re-projection loss_ $L_\text{reproj}$ have very similar effects. They are both enforcing the embeddings to be discriminative. The former goes from 2D pixel to 3D surface and compare the rendered 3D coordinate, whereas the latter also goes from 2D pixel to 3D surface, but then back to 2D pixel (also via rendering) and compare the 2D coordinate. But the ablation results in Table 4 suggests using $L_\text{cs}$ alone without $L_\text{reproj}$ also performs much worse than the full model in some cases. Could the authors elaborate on the difference of the two losses?


## Additional comments
- In Fig 5, it would be more convincing for demonstrating the benefit of aggregating multiple videos, if the multi-video results are also compared to single-video results. It would be even more exciting if the authors can show that this learned category prior enables fast adaptation on new videos or even with still images.
- There is a tiny weird formatting issue in the caption of Fig 3.
- It would be helpful to draw a diagram of the three matching losses $L_\text{cs}$, $L_\text{ct}$ and $L_\text{reproj}$.
- Line 273: "rbreak" -> "break"


---
## Post-Rebuttal
I appreciate the authors' effort in the rebuttal, which has addressed all of my questions. The additional results with higher resolution and comparison between single and multiple videos provide further insights. This is a solid submission with a very interesting approach and great results on an extremely challenging task. I remain strongly in favor of acceptance.
limitations_and_societal_impact: A few failure modes are discussed in the supplementary material, but it can be helpful to also show some visualizations of them.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: VISER tackles the problem of 3D shape and pose reconstruction from an input video with masks and flows. The main competitor on this task is the recent LASR approach (CVPR 2021) The main technical novelty in VISER is joint embedding of pixels and surface points learned with a self-supervised contrastive loss. These embeddings match each surface points to all corresponding pixels in the video, thus addressing long-range correspondences in the video. ViSER outperforms the state-of-the-art on human and animal videos and the supplementary videos are very impressive.
main_review: The paper shows impressive results in the supplementary videos and clearly outperforms the state-of-the-art. The main novelty is the joint pixel and surface embeddings, and how it is used to learn long-range correspondences. The main weakness is writing which is hard to follow.

# Writing :

Though the results are impressive, the quality of the writting could be improved. Some parts are still not very clear to me :

-- Please highlight the similarities and differences between VISER and LASR. Is it reasonnable to think that the core of VISER is the feature embeddings i.e. you could take LASR code and data, add a CNN head to predict per pixel features, an MLP head to predict per surface point features, add the embedding losses and improve over LASR? If so, it would greatly help understand how the paper relates to the literature to clarify this.  

-- Please clarify the exhaustive list of optimized parameters and their shape as done in LASR.’

-- Please clarify equation (8) and (9) with a figure. In general I find the notation R(F) or R(V) uncommon. Maybe it would help to clarify it a bit more beforehand.

-- The words “top-down” and “bottom-up” are used multiple times and confuse me.

--Equation 13 and 14 are not clear. If V_k is a rest shape, then it belongs to R^(N,3) instead of R^3. What is instance k? Do you mean time-step k?

--L191 : “position-and-image-encoded MLPs”→ I don’t know what you mean, a sentence would be better

-- Please add the losses in the pipeline figure. They are critical to get an intuition on why the method works and brings an improvement over LASR

-- Are the motion parameters G_0 … G_B, K represented as the output of a CNN, like in LASR, or explicit parameters of the optimization problem?


# Losses :

I briefly describe the additional losses over LASr as a sanity-check that I understood the approach.

**consistency** : Each 3d point should have the same features as the pixel it projects to.

**constrastive** : each 3D point should follow this cycle : 3d -> projection to pixel space -> estimated corresponding 3D point via cost volume matching. 

**Lreproj** : is the same as the constrastive loss except the cycle is 2d->3d->2d instead of 3d->2d->2d . Why place them in separate paragraphs?

Please clarify what you mean by “Different from geometric cycle consistency loss [19] that suffers from depth ambiguity along the back-projected ray, our formulation leverages graphics rasterization to automatically handle self-occlusion during joint embedding learning.”

# Small comments

**Figure 5:** Some parameters of LASR could be shared across several videos of the same category to compare more fairly with VISER in 4.3 in particular the template shape V and the skinning weights.

**Figure 2 caption :** “Given a <test\> video." There is no training nor testing set in your approach, it is a per-video optimization. Referring to “test” here is confusing.
  
# Feedback  
An additional interesting loss you could consider on the embeddings is enforcing that the distance between two embeddings equals the geodesic distance of the corresponding surface points, as is done in HumanGPS which in their own words “empirically leads to accurate, smooth, and robust results.”

# Post-Rebuttal comment
I have read the other reviews and thank the authors or their responses. I thank the authors for clarifying a number of points that were not clear to me initially. I maintain my initial rating and recommend this paper for acceptance.

limitations_and_societal_impact: N/A
review_title: null

Meta-review: This submission received 4 positive final ratings: 7, 7, 8, 6. The reviewers mostly appreciated novelty (while noting high similarity with LASR), clear presentation and strong empirical performance.
The remaining questions and concerns seemed to be addressed in the rebuttal, as acknowledged by the reviewers.
The final recommendation is therefore to accept as a spotlight.
Decision: Accept (Spotlight)