Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper focuses on the problem of 3D reconstruction of animals from a monocular video. The proposed method is claimed to be Category-agnostic, which means they can deal with different categories such as dogs, horses, but they are all quadruped. The major contributions is the category-agnostic reconstruction which is realized by first retrive a 3D template model from an asset. Then the template model is deformed and optimized to fit to the input video. 
strengths_and_weaknesses: Strengths:
First, having a good initial template to start with the following up optimization will certainly reduce the deformation space and make the optimization to be more feasible. They have demonstrated better visualization results by adopting this retrived template model. The idea is pretty easy to understand and the paper is organized and written well.

Weakness:
1) Obtaining this template model by retrival using the pre-trained CLIP model is more or less an engineering work. I don't think this could be claimed as a technical contribution. The performance is improved mainly due to this retrived template, while the compared approaches start with some general shape for example, a sphere.
2) The optimization pipeline or the loss function is not novel. It is pretty much a standard optimization, I'm a bit confused what the authors want to claim on this optimization problem. In addition, the skeletal representation of articulated models, they are just standard way of dealing with those kinds of animals. I'm not sure what the authors want to emphasize on this.
3) Missing important comparison:  -- BANMo: Building Animatable 3D Neural Models from Many Casual Videos. 

questions: I have some questions or confuse on the some technical details of the proposed method, 
1) How to optimize both the bone length, joints angles, skinning weights together? Do we need to optimize one while fixing others? Furthermore, what is the improvement of optimizing skinning weights, what if the skinning weights are not optimized, but using those from retrived template?

2) How is the optimization initialzed? How can we achieve the initial fitting to images or videos, including initial global pose?

The authors might also want to include the response to the issues I raised in Weakness above.
limitations: The major limitation is lack of technical contribution. Using the pre-trained CLIP model as features to retrive the 3D template is good, but I'm not sure this could be claimed as technical novelty. And the following optimization is also pretty standard. The authors should point out what are the major technical contributions that really stands out.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a category-agnostic character animation reconstrucion from a casual video input. To alliviate the ill-posed nature of the problem, this work first queries the closest template model from the database using CLIP features. As the following inverse graphics optimization stage can warm start with the retrieved template, the proposed approach produces better results both qualitatively and quantitatively over prior methods. Additionally, the paper introduces a large amount of synthetic dataset for qualitative evlauation of predicted attributes with diverse categories.
strengths_and_weaknesses: This paper has the following strengths:
- This work presents an interesting use of CLIP feature. To retrieve the closest animal template from the database, the CLIP features are extracted from both input video and synthetically rendered database. This type of semantic-based retrieval is more general and flexible than hand-crafted descriptors for the retreival task. 
- The paper presents an impressive qualitative results even from videos in the wild. 
- The proposed dataset would be a great resource for community for both training and evaluation to assess the accuracy of predicted attributes from diverse categories. 

There are several weaknesses of this work:
- The qualitative results of baseline methods (LASR, ViSER) are substantially worse than what’s presented in their original papers. I’m wondering if the results are cherrypicked or their code was not properly run. I would highly recommend reaching out to the authors of these papers to confirm that these results are expected. Please answer to this in the rebuttal to prove that the experiments are credible.
- L58-60: I’m very confused about the notion of generalization here. The paper also discuss train/test split in L258. However, as far as I understand, this work presents an instance-specific training and there is no cross-instance generalization. Please clarify.

Overall the paper presents an interesting approach and the results are impressive. However, the aforementioned concerns prevent me from giving a higher score at this stage. 
questions: - Please answer to the comments above.
- It is not clear if CLIP is necessary for this retrieval task as text is not involved at all. It would be great if the use of various feature backbone including CLIP and ResNet pretrained with ImageNet is evaluated. 
- From the exposition, it is not clear how to compute CLIP feature from videos (CLIP only provides embedding per frame). Please elaborate.

Other comments:
- L43: The neural inverse graphics framework in general has been extensively used in prior works, and not novel. Please state more concretely what is novel over the prior works in terms of the optimization.
- L231: Please add citation to Cobra-tools.
limitations: The limitation is discussed, but its societal impact is not.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose CASA, a method for recovering 3D shape and skeletal movements from monocular videos. Given a video, it first retrieves a shape and a skeleton from a library with 200+ animals, then optimizes both shape and articulations with differentiable rendering. Results are shown synthetic real datasets, with applications in re-posing. A new dataset PlanetZoo with 200+ animated animals is introduced.
strengths_and_weaknesses: **Strengths**
- The proposed PlanetZoo dataset is interesting, as I'm not aware of a dataset of similar size (>200 animals) and quality (with texture, skeleton and deformation). It has potential to be used to evaluate and benchmark the performance of animal 3D reconstruction algorithms. From that perspective, it would be helpful to highlight the features of the dataset, and compare with the existing ones (such as deforming things 4D).
- The method is sensible. It leverages 3D shape and skeletal priors of specific categories to improve the reconstruction quality. Since the library is large, the method can be applied to a wide range of animal categories.
- The usage of skeleton structure also helps the re-posing application and user controllability.

**Weakness**
- Quality of results. The results are not faithful to the input data even test-time optimization is used. For instance in Fig.7, the reference cow image does not contain a horn but the reconstruction has horns, possibly from the template being retrieved. This is not desirable. I'm also confused as I did not notice a term that prevents the template shape to change when there is disagreement with the image evidence during optimization. This makes it unclear whether the optimization works properly.
- The experimental comparison can be made fairer. LASR/ViSER does not have access to a 3D shape library. Proper baselines would be methods that use a 3D shape template, such as ACSM [A], ACFM, or proving LASR a 3D template.
- There are some missing details (see questions). The one I'm most concerned about is the initialization of root transformations, which is crucial for reducing the ambiguity between shape and deformation. For instance, if the heading direction is rotated by 180 deg at initialization (the head becomes the tail), the optimization might focus on deforming the shape without correcting the heading.
- Ablation study is missing in the main paper. How well does retrieval perform? As it is a major contribution, a quantitative evaluation and thorough analysis is expected. For the ablation study on flexible bone model, a qualitative comparison is desired as numbers does not give much insight in this context.
- Fig. 3 is slightly misleading as it is not clear which parameters are per-frame and which are per-video. The subtitles also conflated per-video deformation (stretch and deform) vs per-frame deformation (rigging), vertex deformation (deform) vs bone deformation (stretch). 

**Other related work**
- [A] Articulation-aware canonical surface mapping. CVPR 20.
- [B] Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects, CVPR 22.
questions: - How many categories does the dataset contain? What are the categories?
- How are the root node transformations represented and initialized? Are they the same as LASR/ViSER?
- During model retrieval, how to sample viewpoints when rendering the 3D model? How many viewpoints are needed? How long does it take to sampling and matching? Also, pairwise matching should produce SxNxT scores, where S is the number of shapes, and N is the number of renderings per shape, and  T is the number of frames. Then how is T and N marginalized?

limitations: Yes.
review_title: null

Meta-review: metareview: The paper shows how to combine 3d model retrieval with an inverse graphics framework to recover 3D models of a diverse range of animals from video.   The paper also introduces a new dataset of 3D animals that is projected to be of value in future works.

While one reviewer considers the technical problem to be "an engineering work", the other reviewers, and the AC, consider that the implementation and experimentation of the effects of this novel (in this context) idea is valuable.

Based on calibration across other papers and reviews in this AC's stack, the average review score is generally inconsistent with the review text, even given the effective rebuttal.  I mention this only because a poster acceptance might seem at odds with average score, but of course the point of meta reviewing is to make a judgement which looks at more than average score.  The key decision that might be affected in this case is oral vs poster, so it is perhaps useful to clarify: an oral presentation needs to be of value to the broad NeurIPS community.  3D computer vision is an important subfield, and animal reconstruction is an emerging topic in the subfield, but the learnings of this paper remain essentially within a subfield, so I am confident that poster is the appropriate disposition of this paper.

confidence: Certain
Decision: Accept