Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies self-supervised learning of articulated shapes under a very challenging setting where only a few (~30) in-the-wild images of an animal category are available. This setting is much more challenging than prior art as silhouettes and templates are not used and the number of images used for training is very few. To make this possible, the authors propose to model articulated objects with a 3D skeleton, and each part is modeled with a simple shape. The model is trained via analysis by synthesis, where DINO features are used to provide silhouette and 2D-3D semantic consistency. Experiments on Pascal Part and self-collected web images demonstrate that the proposed method LASSIE achieves considerably better 3D reconstructions and 2D&3D part discovery compared to baselines.
strengths_and_weaknesses: Strengths:

1. The proposed method LASSIE is novel and reasonable. The design of modeling each part using simple shape primitives effectively regularizes the solution space and makes the problem more tractable. The 2D-3D semantic consistency loss makes use of the semantically consistent DINO features to help discover 3D parts. The regularization and training procedure are reasonable. I believe these new insights and designs are valuable to the community.  

2. Thanks to the proposed method, it is the first time that the self-supervised articulated shape learning task can be addressed under such a challenging setting where only a few (~30) in-the-wild images are available. LASSIE also considerably outperforms previous methods, and has broad applications such as semantic part refinement, pose/texture transfer.  

3. The paper is well written.  


Weaknesses:

This paper is generally fine. The following weaknesses are not significant.

1. In line 278, it is mentioned that there exists no closer framework other than 3D Safari [39] and A-CSM [17]. However, it seems to me that these are not the state-of-the-art method for self-supervised articulated shape learning. For example, the following papers are later than the two baselines. Please clarify why [39] and [17] are selected instead of these newer ones.  
[1] Tulsiani et al, Implicit Mesh Reconstruction from Unannotated Image Collections, NeurIPS2020.  
[2] Ye et al, Shelf-Supervised Mesh Prediction in the Wild, CVPR2021.  
[3] Li et al, Self-supervised Single-view 3D Reconstruction via Semantic Consistency, ECCV2020.  

Typos and grammar issues:  
Line 12: We -> we.  
Line 146: k denotes -> b denotes.  
Line 198: The use of ... -> Through the use of ... or By using ...  
Line 223: a image -> an image.  
Line 286: an object category is able to produce detailed shapes of animal bodies which -> an object category which is able to produce detailed shapes of animal bodies.  

questions: Please clarify why [39] and [17] are selected instead of the newer ones as mentioned in weaknesses.
limitations: Yes, the authors adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an optimization-based methods that reconstructs 3D part based articulated shapes of animals from only a small collection of in-the-wild images (~30 instances) of different instances of an articulated object category, eg, horse, giraffe, elephant, penguin etc. The key idea is leverage self-supervised pretrained ViT features (DINO) to establish semantic correspondences between the images of different instances with various camera viewpoints, pose articulations, texture and environment. By constraining the 3D shapes with a set of pretrained part primitives (ellipse, cylinder, cone) on a pre-defined 3D skeleton (eg, quadrupedal or bipedal), the model is able to reconstruct compelling articulated 3D shapes of different instances, significantly better than existing learning-based models trained on only one specific category (zebra or horse).
strengths_and_weaknesses: ## Strengths
### S1 - Compelling results on an interesting and challenging task
- The task of reconstructing articulated 3D animals from only a small collection of in-the-wild images of various articulated instances is highly ill-posed and very challenging. The proposed method seems to work well on a diverse set of examples (9 different animals demonstrated).
- Based on the examples shown in Fig 9 in the supplementary material, the images in each category collection are also very diverse in terms of camera viewpoints, poses, and environment and illumination conditions.

### S2 - Good ideas and careful implementations
- There are two main ideas. One is to leverage self-supervised pretrained image features (DINO ViT) to establish semantic correspondences between different image instances via simple clustering. This is more powerful than previously studied SCOPS which still requires ImageNet pretraining and large training datasets.
- The second idea is to use part-based 3D representation with a pre-defined generic 3D skeleton. This significantly reduces the complexity of this ill-posed task, and yet requires only minimal priors.
- These two ideas are carefully implemented and validated through ablation studies. In particular, the use of an neural surface representation and VAE pretraining of the 3D parts seems quite effective in achieving more regularized shapes.

### S3 - Good writing
- The paper is very well written, with clear motivations, sufficient technical explanations and illustrative visualizations.


## Weaknesses
### W1 - Optimization-based method
- The method optimizes over a small collection of images and does not generalize to new images out of the box. It would also be interesting to extend the idea to a learning-based pipeline that would allow inference on novel images.
- This also makes the comparison against previous learning-based methods unfair, as the results of other methods (I assume) are obtained by one forward pass without any finetuning. It would be slightly fairer to also perform some test time finetuning on the other methods.

### W2 - Heavy regularization
- Due to the ill-posedness of the task, the model relies on a number of heavy constraints, including a simplistic part-based shape representation which also requires pretraining using a set of primitives. This largely limits the expressivity of the model and hence does not lead to fine geoemtric details.
- The need of hand-crafted 3D skeleton is also a major limitation. Although it may seem effortless on one category, it is still quite cumbersome to scale up to all kinds of objects. Moreover, the model only works well on animals that have limited articulated poses, where the skeleton is clearly visible. I can imagine that it would still struggle for animals that highly articulated, like cats (in which case, only a small number of images may not be sufficient).
- There are also a number of specific constraints in the model, eg, minimizing the sideway rotation angles on the leg bones, and a specific optimization procedure that alternates between viewpoints, bones, parts and features.


### (Minor) Clarifications
- Are the part prior latents $\mathbf{e}_i$ randomly initialized and optimized for each part during full training (after VAE pretraining)?
- I am confused by the visualization of the part primitives in Fig 2. Aren't the output of the neural surface networks $\mathcal{F}_i$ supposed to be elongated shapes in the canonical space (without bone orientations)? Fig 2 seems to suggest that the part shapes predicted by the networks are already oriented.
- Line 257: the optimization procedure seems to involve three steps: 1) camera viewpoints, 2) bone transformations, 3) latent part codes and part deformations. Is that correct? What about the 3D part features? When are they updated (the E step)?
- It seems pretty easy for the model to confuse head and tail by looking at only the 4 clusters visualized in the zebra example in Fig 2. Since camera viewpoint is optimized in the first step, will it get confused initially and fall into local minima. And if so, how often does it occur?
- Were the 2D keypoints used in keypoint transfer evaluation manually labelled?


### (Minor) Typos
- Line 146: I believe $k$ should be $b$.
- Line 171: add $t^j \in \mathbb{R}^{b \times 3}$ for completeness. Also, make the notations consistent -- $\mathbf{t}$ vs. $t$.
- Line 286: fix typos.
questions: Apart from a few questions that need clarifications listed above, the only additional result I might suggest is to also perform test-time optimization with previous methods for a slightly fairer comparison, maybe in the final version. The settings are different enough though, so this may not really affect the evaluation of the paper.
limitations: The paper has included a discussion on the limitations of the model as well as illustrative examples in the supplementary material.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a new task formulation in the area of non-rigid reconstruction and parts discovery. The model is fit to a small collection of images (25–30) of unrelated instances of a category (such as animal species). It enables discovering 3D pose and shape of objects in all images, along with part segmentation. No supervision on the level of images is required, however the users have to provide the topology and a typical pose of the skeleton, assignment of image parts to image features’ clusters, and also there are specific constraints on the movement of certain joints (such as legs in case of quadrupeds). The model produces better PCK numbers than competing methods (Articulated CSM and 3D Safari) and visually convincing reconstructions on the examples shown.
strengths_and_weaknesses: Originality:
* I see the method as an extension of UMR [19] to articulated shapes, which are naturally represented as a union of convex’ish primitives, where SCOPS features are upgraded to DINO. Extending to articulated shapes requires overcoming some technical challenges, such as adding regularisations and priors.

Clarity:
* (+) In general, the paper is well written, and the method is mostly clear,
* (−) some of the limitations are either not addressed or not mentioned in the introduction (such as the required category-level supervision), while l. 51 claims that the method is “completely self-supervised”,
* (−) I am confused about the final optimisation algorithm; it runs block-coordinate optimisation (l. 256 and on), but then there is EM-like optimisation of features inside one of the steps; I think the paper would benefit from pseudo-code of the training loop,

Significance:
* (+) getting rid of instance-level supervision is a great step; the method can scale to larger collections easily;
* (+) the required prior information is indeed easier to obtain than e.g. category-specific mesh.

Experiments:
* (+) visual demonstrations look good;
* (−) is the EM-like optimisation important? What would be the result if the original DINO features were used in (4)? (I think it corresponds to 1 iteration of “EM”);
* (−) evaluating using 2D keypoint transfer may be used by other papers but it does not tell much about reconstruction quality; representing the animal’s shape with a flat surface would result in a relatively good PCK while such reconstruction is visually unacceptable. I suggest evaluating on datasets where 3D ground truth is available, such as synthetic datasets or human datasets (such as Human 3.6M) – without expectation of performing better than methods specialising on human reconstruction,
* (+) Applications paragraph/results are nice.

=============

Typos:
* line 80: In contrasted ← in contrast;
* line 146: *k* ← *b*;
* line 211–213: why is it called EM-like algorithm? I would call it block-coordinate descent; for EM, I would expect having an explicit likelihood function depending on latent random variables, and taking expectation over those at E-step,
* line 285: “A-CSM assume high-quality skinned model”: IIRC, it only requires the mesh segmented to parts but no skinning weights, which is a much lighter assumption.
* line 286: “animal bodies which”.

UPD. The other reviewers pointed me towards the supplementary video that I had previously overlooked. It reassured me in the reasonable reconstruction quality, at least on the shown examples. I like the new problem setting and the method. My only remaining (but major) concern is the presentation and positioning of the method and the results: in the current manuscript, the relation to UMR (which is strong, at least on the ideas level) is essentially ignored; Experimental section wrongly claiming conclusive quantitative evaluation of 3D reconstruction; the limitations are not addressed, and the Introduction / abstract massively oversell the method (which it ironically does not even needs). I am changing my rating to borderline to not block the acceptance; I will thus leave it to ACs to decide whether we can accept the paper in this form.
questions: Major issues:
* I suggest to allocate more text to position the method w.r.t. UMR [19] more clearly;
* I suggest to evaluate reconstruction quality on datasets with 3D ground truth (see above);
* can you clarify the comparison protocol with the baselines: are they evaluated on the hold-out evaluation set (vs. the proposed method that fits the model to all available images as far as I understood)? 
* please address the limitations below.

Other issues:
* In Chamfer loss (3), is the closest point chosen using the distance (4) rather than just geometric distance? If so, what is the motivation of that? I feel like the loss should enforce points to reproject in the correct locations; matching the features should not compensate for that.
* in Experiments, baselines sometimes are trained using a different category; please indicate clearly in the results table where it is the case,

limitations: The paper lists some limitations, but crucially misses some others:
* Can the method generalise to new images, i.e. can it infer the pose for a new instance without re-fitting the model? The competing methods only need to run the forward pass. That may be important for interactive applications.
* The method uses the same shapes of the individual parts across instances (ll. 172–174), so the resulting meshes differ only in the parts’ rigid transformations. This is an approximations since in reality individuals differ in the bone lengths and deformation of their flesh.
* Another consequence of that is, if I understand correctly, reposing the shape can result in gaps between the parts. In fact, lack of those gaps is guaranteed for training instances only thanks to rendering losses; this means that the invisible surface may not be represented properly,
* the loss L_pose penalises the difference from the rest pose; it is usually difficult to find the weight of such loss so that the method is able to fit extreme poses,
* the need for heuristics like L_ang (5) limits the applicability of the method to new categories where users would need to figure out similar constrints.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The submission propose Lassie to obtain articulated 3D reconstructions of a skeleton-based animal class (shown are quadrupeds and bipeds) from an image collection of 20-30 images of different individual animals. Given a fairly generic animal skeleton (16 vertices and 15 edges), it learns a 3D part for each edge/bone. The 3D part is represented by regressing 3D offsets for a 3D unit sphere via a coordinate-based MLP. The shapes of the part are shared across instances, only a similarity transform is applied based on the skeleton of each instance. The method requires only little outside knowledge: It clusters DINO-ViT [5] image features to obtain a very rough 2D part annotation of the images, and it requires manual matching between these part annotations and the skeleton parts. The optimization determines instance-specific skeleton parameters and class-specific part parameters. The main loss is a novel semantic matching loss that has some similarity to the one used in BanMo since it transfers image features to the 3D reconstruction and then tries to match the reconstruction to the images using those features. 

There is no prior work that tackles this problem setting. Comparisons to A-CSM and 3D Safari show better results for Lassie, in several distinct settings.
strengths_and_weaknesses: =====STRENGTHS=====

The problem setting is interesting and it is remarkable that the proposed method takes only 10 minutes on a consumer GPU to obtain quite decent articulated reconstructions from only 20-30 images of an animal class. This opens up the door for a lot of follow-up work and raises a bunch of interesting questions, although unfortunately code will not be released.

While there are an MLP for the part shape and DINO image features, the majority of the method is not much concerned with neural networks, which is a nice change of pace.

The method is not exactly simple but also far from being convoluted. Its complexity seems appropriate to me and the design choices are argued for in the paper. 

The evaluation is done well, including additional experiments in the supplement.

The paper is well written. The presentation is nice, including clear tables and figures.

I appreciate the name of the method.

=====WEAKNESSES=====

There are no significant weaknesses I can see. I do have a number of questions and suggestions though, see below.

Minor writing issues:
* L.12: "We" should be lower case
* L.30: "a" -> "the"
* L.146: "k" -> "b"
* L.286: sentence ends in "which"

Related Work:

* Missing related works that are from the same line of work as A-CSM [17]: Kanazawa et al. Learning category-specific mesh reconstruction from image collections (CMR; ECCV 2018) and its follow-up works, see e.g. Table 1 in Wu et al. DOVE: Learning Deformable 3D Objects by Watching Videos (arxiv preprint 2021):

- Kulkarni et al. Canonical surface mapping via geometric cycle consistency (CSM; ICCV 2019), 
- Goel et al. Shape and viewpoints without keypoints (U-CMR; ECCV 2020), 
- Li et al. Self-supervised single-view 3D reconstruction via semantic consistency (UMR; ECCV 2020), 
- Li et al. Online adaptation for consistent mesh reconstruction in the wild (VMR; NeurIPS 2020),
- Kokkinos et al. Learning monocular 3D reconstruction of articulated categories from motion (CVPR 2021),
- Kokkinos et al. To The Point: Correspondence-driven monocular 3D category reconstruction (NeurIPS 2021)

* Less crucial, several animal shape estimation methods that use SMAL or similar statistical models are not cited:

- Biggs et al. Creatures great and SMAL: Recovering the shape and motion of animals from video (ACCV 2018)
- Biggs et al. Who left the dogs out: 3D animal reconstruction with expectation maximization in the loop (ECCV 2020)
- Badger et al. 3D bird reconstruction: a dataset, model, and shape recovery from a single view (ECCV 2020)
- Wang et al. Birds of a feather: Capturing avian shape models from images (CVPR 2021)

* The early important work of Cashman et al. What shape are dolphins? Building 3d morphable models from 2d images (TPAMI 2013) is also not cited.
* Another work that reconstructs dynamic objects from in-the-wild image collections is Wu et al.'s Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild (CVPR 2020 best paper).
* While NeRF and a couple static variants are mentioned, the NeRF field has also produced some works on dynamic reconstruction of general objects from video (not image collections; although see Banmo). These could be added for completion, see e.g. the recent survey Tewari et al. Advances in Neural Rendering (Eurographics 2022).


---- post rebuttal

My rather minor concerns and questions have been addressed well. I increase my rating to accept.
questions: Questions

* The semantic loss is interesting. It would be nice to see a discussion on how it differs from BanMo's loss (see summary above).
* I would like the authors to comment on why comparisons against CMR, CSM, U-CMR, VMR, etc. (see Weaknesses) is not necessary, e.g. because there is simply no code. 
* (Almost?) all methods that follow CMR (reconstructing animals/dynamic objects from image ensembles) evaluate on the CUB-200-2011 birds dataset. Why is there no evaluation on birds? Is there too much variablility? A-CSM [17], against which the submission compares, also evaluates on this dataset.
* The checklist states that error bars are provided. But I do not see any numbers or any other indication that multiple runs/seeds were used anywhere. Am I overlooking something or is this answer in the checklist wrong?

* While not crucial, the ablation on the number of images per animal in supplemental Sec. 2.2 uses the zebra class, which seems like a quiet easy class (see e.g. Fig. 9 in the supplement; all zebras have a pretty similar pose). I am curious how well a reduced number of images would work for giraffes or kangaroos or maybe horses, especially when keeping diverse poses and appearances in that subset of images.

Suggestions (do not need to be addressed in the rebuttal but should be considered for a revised version of the paper):

* I am not really satisfied with saying that no annotations are used (e.g. L.166). The DINO features give rise to 2D part segmentation masks. DINO is self-supervised, but still. 
* While I find the evaluation sufficient already, results on humans would be interesting. 
* The conclusion claims that no human annotations (L.342f) is used, but that's not correct (L.231ff). Please make sure that claims about annotations are precise everywhere.
* Please provide the mathematical formulation of L_lap and L_norm (and all other terms that might not be defined) in the supplementary material.
* In Fig. 10 and 11 in the supplemental document, please make the novel views of the same size as the input views.
* The current uniformly colored visualization of the parts shows coarse-level correspondences across instances. It could be interesting to color each part with some kind of pattern (e.g. a rainbow or color gradient) to visualize fine-level correspondences.
* L.167f says that the instances can have "varying texture (appearance) due to pose, camera angle, and lighting". What about the individual identity, i.e. different individuals have different fur? The method seems to not have an issue with it, I think it should be added to that list.
* L.256ff state that alternative optimization of camera, pose, and shape is used. Does that mean that the camera is optimized at the beginning of training and then never again? Or is it camera, then pose, then shape, then camera, then pose, then shape, etc.? If it's the former, maybe calling it training stages instead of alternative optimization would be more descriptive.
* What's the value of m in L.215?
* Please render the images in a higher resolution. They are quite pixelated already.
limitations: The checklist states that potential negative societal impact is discussed. While I don't see any such potential negative impact, I don't believe that that is discussed anywhere in the paper, despite what the checklist says.

Limitations are discussed, including failure cases in the supplement.
review_title: null

Meta-review: metareview: This paper had substantial discussion amongst reviewers, and concluded with mixed reviews (7, 7, 7, 4). The positive reviewers (mH51, rsf7,  LjEQ) actively and strongly championed the paper. The remaining concern comes from boNQ, who (in reviewer-to-reviewer discussions) was primarily concerned with making sure that the authors are clear about specifying limitations in method and evaluation. In particular, boNQ was concerned about making sure the following aspects were clearly and prominently discussed in the paper: (a) generalization to new images; (b) assumption of the rest-pose prior; (c) lack of adjusting the shape per-instance; and (d) non-guarantee of gaps between parts. Additionally, boNQ  was concerned about keypoint-based evaluations as a proxy for 3D (although was convinced by the videos in the supplemental). 

The AC has examined the paper, reviews, and discussion, and is inclined to agree with the accepting reviewers. The paper will be a strong contribution to the literature and will be of great interest the community. However, the AC notes that many of the reviewers may have similar questions to boNQ. Thus, the AC strongly encourages the authors to address boNQ's concerns in the final version of the paper. With the additional space, the AC believes it will be feasible to specify the limitations more clearly and earlier in the manuscript, and also somehow incorporate additional visualizations demonstrating the effectiveness of the reconstructions. While there is no mechanism for enforcing these changes, they will substantially improve the reception of the paper. 
confidence: Certain
Decision: Accept