CASA: Category-agnostic Skeletal Animal Reconstruction
Authors: Yuefan Wu
Zeyuan Chen
Shaowei Liu
Zhongzheng Ren
Shenlong Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: articulation
inverse graphics
3D reconstruction
animation

Abstract: Recovering a skeletal shape from a monocular video is a longstanding challenge. Prevailing nonrigid animal reconstruction methods often adopt a control-point driven animation model and optimize bone transforms individually without considering skeletal topology, yielding unsatisfactory shape and articulation. In contrast, humans can easily infer the articulation structure of an unknown character by associating it with a seen articulated object in their memory.  Inspired by this fact, we present CASA, a novel category-agnostic articulated animal reconstruction method. Our method consists of two components, a video-to-shape retrieval process and a neural inverse graphics framework. During inference, CASA first finds a matched articulated shape from a 3D character assets bank so that the input video scores highly with the rendered image, according to a pretrained image-language model. It then integrates the retrieved character into an inverse graphics framework and jointly infers the shape deformation, skeleton structure, and skinning weights through optimization. Experiments validate the efficacy of our method in shape reconstruction and articulation. We further show that we can use the resulting skeletal-animated character for re-animation. 


CASA: Category-agnostic Skeletal Animal
Reconstruction
Yuefan Wul Zeyuan Chen
Shaowei Liu University of Science and Technology of China
Zhongzheng Ren
Shenlong Wang?
University of Illinois Airana-Champaige
https://Iven Wu. gi github.... io/CASA
Input
Re-animation
(unknowa animal)
Shape
Skeleton
Rigging
Novel pose
Figure 1: Given monocular video with an animal from unknown category, CASA jointly infers the articulated skeletal shape and rigging through optimization, which can be animated into novel poses.
Abstract
Recovering the skeletal shape of an animal from a monocular video is longstand- ing challenge. Prevailing animal reconstruction methods often adopt control-point driven animation model and optimize bone transforms individually without consid- ering skeletal topology, yielding unsatisfactory shape and articulation. In contrast, humans can easily infer the articulation structure of an unknown animal by as- sociating it with seen articulated character in their memory. Inspired by this fact, we present CASA, novel Category- Eategorr-Agnanne Agnostic Skeletal Animal reconstruction method consisting of two major components: video-to-shape retrieval process and a neural inverse graphics framework. During inference, CASA first retrieves articulated shape from 3D character assets bank so that the input video scores highly with the rendered image, according to pretrained sanguage-vision model. CASA then integrates the retrieved character into an inverse graphics framework and jointly infers the shape deformation, skeleton structure, and skinning weights through optimization. Experiments validate the efficacy of CASA regarding shape reconstruction and articulation. We further demonstrate that the resulting skeletal- animated characters can be used for re-animation.
Introduction
Recovering the shape, articulation, and dynamics of animals from images and videos is a long- standing task in computer vision and graphics. Achieving this goal will enable numerous future applications for 3D modeling and reanimation of animals. Nevertheless, accurately reasoning about geometry and kinematics animals the wild remains an ambitious problem three reasons: 1) partial visibility of the captured animal, 2) variability of shape across different categories, and 3) ambiguity of unknown kinematics. Take the camel in Fig. as an example to faithfully model requires: 1) ballucinating the unobserved (e.g.. occluded, back-side) region, 2) recovering its unique shape and scale, and 3) predicting its kinematic structure.
Remarkable progress has been made in addressing the aforementioned challenges by either exploiting
richer sensor configurations (e.g., multi-view cameras [11, 17] depth sensors [41, 42, 50]), or by
Indicates equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

making strong class-specific assumptions (e.g., humans [34, 55-57, 78, 80] or quadruped animals [83- 85). However, these assumptions greatly limit the applicability of reconstruction systems, which fail to generalize to animals in the wild.
We aim reconstruct arbitrary articulated animals using monocular videos casually captured in the wild. Recent works [74-76, 70, 45] demonstrate promising results. However, they often impose non-realistic assumptions on articulation, such as control point driven deformation [74-76, 70] or free- form deformation [45, 50J. As a result, they fall short of the goal of modeling skeletal characters that can be realistically re- e--animtedd in downstream applications. Furthermore, there remains significant improvement space for the quality of the inferred animal shape.
In this work, propose CASA, a novel solution for Sategory-Agnosiie Skeletal Animal recon- struction in the wild. CASA jointly estimates an arbitrary animal's 's 3D shape, kinematic structure, rigging weight, and articulated poses of each frame from a monocular video (Fig. 1). Unlike existing nonrigid reconstruction works [74, 75, 70J, we exploit a skeleton- articulation model as well as forward kinematics, ensuring the realism of the resulting skeletal shape ($ 3.1). Specifically, we propose two novel components: a vidco-to-shape retrieval process and a ekeleton-basee neural inverse graphics framework. Given an input video, CASA first finds template shape from a 3D character assets bank so that the input video scores highly with the rendered image, according to a pretrained language-vinn vision model [47] ($ 3.2). Using the retrieved character as initialization, we jointly optimize bone length, joint angle, 3D shape, and blend skinning weight so that the final outputs are consistent with visual evidence, i..., input video ($ 3.3).
Another issue that hinders the study of animal reconstruction is that the existing datasets [4, 13,31] lack realistic video footage and ground-truth labels across different dynamic animals. To address it, we introduce a photo-realistic synthetic dataset PlanetZoo, which is generated using the physical- based rendering [7] and rich simulated 3D assets [35]. PlanetZoo is large-scale and consists of 251 different articulated animals. Importantly, PlanetZoo provides ground-truth 3D shapes, skeletons, joint angles, and rigging, allowing evaluation of category-agnosta 4D reconstruction holistically (S 4).
We evaluate CASA on both PlanetZoo and the real-world dataset DAVIS [46]. Experiments demon- strate that CASA recovers fine shape and realistic skeleton topology, handles a wide variety of animals, and adapts well to unseen categories. Additionally, showed that CASA reconstructs skeletal-animatable character readily compatible with downstream re-animation and simulation tasks.
In summary, we make the following contributions: 1) We propose simple, effective, and gener- alizable video-to-shape retrieval algorithm based on a pretrained CLIP model. 2) We introduce a novel neural inverse graphics optimization framework that incorporates stretchable skeletal models for category- agnostic articulated shape reconstruction. 3) We present a large-scale yet diverse skeletal shape dataset PlanetZoo.
2 Related work
Category-specifie nonrigid reconstruction. Parametric model (template) is a classic approach for category-specifie nonrigid reconstruction(440, 18, 2, 78, 77, 36, 32, 54] Templates are built for various objects including human body [34, 1]. hand [53]. face [29, 3], and animals [85, 2]. Template- frre methods [19, 30, 25, 64, 57, 70, 23, 44, 69] study the problem of predicting nonrigid objects directly for a specific category. However, these methods heavily rely on strong category priors such as key-points annotations [19], canonical shapes [10], temporal consistency constraints [30], or canonical surface mapping [23], making them hard to generalize to category-agnostee setting.
dategory-agnostie nonrigid reconstruction. Classic approaches [S, 24, 60, 43] utilize Non-Rigid Structure from Motion (NRSfM) for non-rigid reconstruction, which requires point correspondences and strong priors shape motion. However, NRSfM does not work well for monocular videos in the wild, in which obtaining long-range correspondences is hard. Recent works adopt learning-basee approaches via either dynamic reconstruction [27, 50] or inverse graphics optimization (74-76).
However, these methods do not reason skeletal kinematics and suffer from incorrect shape prediction when certain parts are occluded or invisible.
2

---

Skeletal Charatto
b
Wak
qi
Shape
Skaleton
Rigging
A Artiqilated Pose
Porward Kinematics
Opimizatioon
CL(P-gudded Retrieval
Init Character
Newsl
Energy Evaluation
Final Skeletal Stape
Pretrinedd
Flow Loss
Silhouette Loss
Symmetry
Smoothness
Input Video
Figure 2: Overview. Given an imput video, A video-to iidoott--saape retrieval process is first conducted with the guidance of pre-trained CLIP ($ 3.2). Initialized by the retrieved shape, we jointly optimize shape, skeleton (bone length and joint angle), and skinning through inverse rendering ($ 3.3).
Animatable shape. Humans and animals undergo skeleton-driven deformation. A skeletal shape often consists of two parts: a surface mesh representation used draw the character and skeleton structure to control the motion. Each skeleton is hierarchical set of bones. Each bone has associated with a portion of vertices (skinning). The bone's transformation is determined through forward kinematics process. As the character is animated, the bones change their transformation over time. Linear Blend Skinning (LBS) [37, 28] is standard way for modeling skeletal deformation, which deforms each vertex of the shape based on linear combination of bone transformations. For improvement, multi-weight enveloping [68, 38] is used overcome the issue of shape collapse near joints when related bones are rotated or moved. Dual-quaternion blending (DQB) [20] adopts quaternion rotation representation to solve the artifacts in blending bone rotations, and STBS [15] extends LBS to include the stretching and twisting of bones. Recent works [74-76] employs LBS for modeling the motion of shapes recovered from video clips. However, these methods do not enforce skeleton-based forward kinematic structure. Hence their recovered animated shapes are not interpretable and cannot be directly used in skeletal animation and simulation pipeline.
Language-vision approaches. Self-supervised language-vision models have gone through rapid advances recent years [62, 66, 48] due their impressive generalizability. The seminal work CLIP [66] learns a joint lunguage-vision embedding using more than 400 billion text-image pairs. The learned representation is semantically meaningful and expressive, thus has been adapted to various downstream tasks [82, 71, 49, 79, 65]. In this work, we adopt CLIP in the retrieval process.
3D reconstruction dataset. A plethora of synthetic 3D datasets [61, 81, 16, 52] and interactive simulated 3I) environments [58, 59, 9, 67] have been proposed in recent years. ShapeNet [8] provides benchmark for common static 3D object shape reconstruction. Among all of these datasets, only few aim for dynamic object reconstruction [31]. Given that the synthetic dataset has a large domain gap to real-world settings, photo-realistic rendered samples are strongly preferred. To this end, propose photo-realistic synthetic dataset PlanetZoo to study the dynamic animal reconstruction problem. PlanetZoo contains
assets and covers a wide range of animal categories.
3 Category-agnostic skeletal animal reconstruction
CASA aims to reconstruct various dynamic animals in wnld from monocular vidcos. It takes as input the RGB frames {Ic )...r from a monocular video, object masks {Mr) I..TT. and optical flow maps {F, J1..r computed from consecutive frames. From these inputs, we aim recover animal shape Sy and its articulated deformed shape s, at each time t.
Fig. demonstrates an overview of our approach. Our method exploits the skeletal articulation model and forward kinematics to ensure the realism the resulting skeletal shape ($ 3. Our method consists of two phases. In the retrieval phase ($ 3.2), CASA finds template character from an existing asset bank based on the similarity the input video in the deep embedding space using pre-trained encoder. The retrieved template is fed into the neural inverse graphics phase as

---

Per-video parameters
Per-frame parameters
bo
b
wa
b1 t
Wal
91 1
Vi
Vi Skeletal parameters
Deform
Stretching
Rigging
Figure 3: Skeletal shape parametric model. Our parametric model consists of joint angles q. bone length b. skinning weight w, and vertex positions v. Joint angles change per-frame and the others are universal. Utilizing this model, animal shapes are tuned by vertex deformation as well as the stretching of bones. The target articulation would be fit by predicting feamfr joint angles.
initialization(S 3.3). Finally, we jointly reason the final shape, skeleton, rigging and the articulated pose of each frame through an energy minimization framework.
3.1 Articulation model
Skeletal parameters. We exploit a stretchable bone-based skeletal model for our deformable shape parameterization ,as shown in Fig. This articulated model consists of three components: triangular mesh consisting of set of vertices and shapes in the canonical pose, describing the object's shape: 2) a bones connected by a kinematic tree. Each one has a bone length parameter and associated rigging weight over each vertex; 3) joint angle describing the relative transformation between each adjacent bone. To summarize, a deformed shape S; at time can be represented as:
S: E R&j (f, E {w,, E [A]] E fbx E ER') :
{ E (0(3))
(I)
where v; is the vertex position, f, represents triangle face parameterized as a tuple of three vertex indices, W; is the rigging weight constrained in K-dimensional simplex: A (wo...., wK) E R Et Wy and wi 0]: by is the scalar bone length for each bone and q is the joint angle for each bone time t, represented as unit quaternion in the SO(3) space. Note that all the variables except joint angles are shared across time.
Our skeletal model is grounded by the nature of many articulated objects. Unlike the commonly used control-pont point based articulation [41, 75, 74], our model reflects the constraints imposed by bones and joints, leading to more natural articulated deformation. Compared to category-speciff parametric models [34, 85], it is more flexible and generalizable.
Forward kinematics. We use the forward kinematic model [26] to compute the transformation of each bone along the kinematic tree. Specifically, the rigid transformation of a bone is uniquely defined by the joint angles and bone length scales of the bone itself and its ancestors along the kinematic tree. Given bone k. the rigid transform between its own frame and the root can be computed by recursively applying the relative rigid transformation along the chain:
T (Aimaay bansay) = T pa(k) Tapp (qk, by) = II
T pp (qfes bwr)
krEm)(()
Tt where Ti pa(k) is the transformation of bone & and its parent node at frame quedoo busat are the joint angles and bone lengths from all the ancestors of target bone kt T p, (qf. by) is the relative rigid transform between the bone and its parent frame, consists of joint center translation along axis decided by the bone length b = [0, O, by] and a rotation around the joint center R(q()).
Linear Blend Skinning (LBS). Given the rigid transformation of each bone, we adopt LBS to compute the transform of each point. Specifically, the deformation of each point v is decided by
linear composing the rigid transforms of bones through its skinning weight: v = Σ Wi. T where w,, is the rigging weight. T is the transformation of bone k at frame defined in Eq. 2. For simplicity, we omit its dependent variables.
Stretchable bones. The aforementioned nonrigid deformation assumes independent shape and bone lengths. Naturally, the shape should deform accordingly as bone length changes. To model such relationship, exploit stretchable bone deformation model [15]. For each bone, compute an

---

additional scaling transform along the bone vector direction based on its bone length change and apply this scaling transform to any points associated with this bone. Fig. 3 provides an illustration of stretching. For mathematical details on stretching, we refer readers to our supplementary document.
Stretchable bones bring a two-fold advantage: 1) it allows same topological structures; 2) it makes the topological structure adjustable by "shrinking" the bones, e.g., quadrupedal animal can be evolved into seal-like skeletal model through optimization.
model shape variations within the
Reparameterization. Optimizing each vertex position offers flexibility yet might lead to undesirable mesh due to the lack of regularization. We use neural displacement field to re- re-parameterize the vertex deformation to incorporate regularization, such as smooth and structured vertex deformation. We use coordinate-baset multi-layer perceptron (MLP) to define this displacement field V. In addition, we incorporate a global scale scalar u in our framework to handle the shape misalignment between our initialization and the target. The position of each vertex defined as:
V uv; + Vo(uv;)
where uv; is the scaled position at the canonical pose, Vo (epu:))eereeennt represents the vertex offset parameter- ized by 8, and v is the updated position for vertex i. During inference, instead of directly optimizing Vi, we will only optimize the parameters of the displacement network. This displacement field repa- rameterization allow to smoothly deform canonical shape during inference. In practice find this implicit regularization compares favorably over explicit smoothness terms such as as-rigid-as-possibe and laplacian smoothness.
3.2
Video-to-shape retrieval
Directly optimizing all skeletal parameters defined in Eq. 1 is challenging due to the variability and highly structured skeletal parameterization. Many deformable objects share similar topology structures, albeit with significant shape differences. To address it, we propose initializing skeletal shapes in a data-driven manner by choosing the best matching template from asset bank.
Videos and skeletal shapes are in two different modalities; hence establishing similarity measure is hard. Inspired by the recent success of language-vision pretraining models for 3D [66, 39j, we utilize realistic rendering and pretraining image embedding models [47] bridge this gap. Specifically, we first pre-render video footage for each character in the asset bank through a physically based renderer [7]. The environment lighting, background, articulated poses, and camera pose for each video are randomized to gain diversity and robustness. We then extract the image embedding features for each video using CLIP [47], which a language-visn -viiin embedding model that is pretrained on a large-scale image-caption dataset. It captures the underlying semantic similarity between images well despite the large appearance and viewpoint difference. This is particularly suitable for retrieving shapes with similar kinematic structures, as the kinematics of animals is often related to semantics.
During inference, we extract embedding features from given input video and measure the L2 distance between the input video and the rendered video of each object in the embedding space. We then select the highest-scoring articulated shape as the retrieved character. Please refer to the supplementary material for implementation details.
3.3 Neural inverse graphics via energy minimization
We expect our final output skeletal shape to 1) be consistent with the input video observation; 2) reflect prior knowledge about articulated shapes succh as symmetry and smoothness. Inspired by this, we formulate ours as energy minimization problem.
Energy formulation. We exploit visual cues from each frame including segmentation mask {M,) and optical flow {F.} through ffffthh-shelf neural networks [63, 74]. We also incorporate two types of prior knowledge, including the motion smoothness for joint angles qf of bones across frames, as well as the symmetry constraint for the per-vertex offset v; at the reference frame. In particular, let = ((I,), (M,), (F.)) be the input video frames and corresponding visual cues, and (s:) } be the predicted shapes of all frames, we formulate the energy for minimization as:
min X ((s:), + X2 ((xm) + Xa Eryman ((s:))
(s.h...
where Pree measures the consistency between the articulated shape and the visual cues; Fanoceth promotes smooth transitions overtime; Eymm encodes the fact that many articulated objects
5

---

symmetric. The three energy terms complement each other, helping our model capture motions of the object according visual observations, as well constraining the deformed object be natural and temporally consistent. We describe details of each term below.
Visual consistency. The visual consistency energy measures the agreement between the rendered maps of the articulated shape and 2D evidence (flow and silhouettes) from videos. We use differentiable renderer [33] generate projected object silhouettes {M:). Additionally, we project vertices of predicted shapes at two consecutive frames to the camera view and compute the projected 2D displacement to render optical flow, following the prior work [74]. We leverage PointRend [22] for object segmentation and volumetric correspondence net [73] for flow prediction. The energy
measures the difference between the rendered and inferred cues in Eg distance:
Exue = E {I ( + B| - Tao (st)])
(5)
where 3 is the trade-off weight; Thow (s;) is the rendered the 2D flow map for each pixel given the deformed shape 8;; Thy (S:) is the rendered object mask. Similar to previous inverse rendering work [75], the object-camera transform is encoded as the root node transform.
Motion smoothness. This energy term encodes that motions of animals should be smooth and continuous across frames. We impose a constraint to ensure that there is little difference in joint angles of one bone from Two consecutive frames should be slight. We implement this by computing the multiplication of the joint quaternion at the current frame and the inverse of the joint quaternion the next frame, which should be close an identity quatermion q (0, O, 0, 1):
Emmothh Σ>
Σ ([qbt)
qk (+1 -1l2
(6)
where o is the quaternion composition operator.
Symmetry offset. This term encourages the resulting shape in canonical space be symmetric the reference frame. It is inspired by the fact that most animals in the real world are symmetric if putting them into a certain canonical pose (e.g.. "T-pose" for bipedal animals). Following previous works [74], we enforce this property at the canonical shape (where the joint angles are all zero). Specifically, we calculate the chamfer distance for measuring the similarity between the set of vertices {v;} under the canonical shape and its reflection:
Enymm
Ceram
((v.))
where is the Householder reflection matrix.
3.4 Inference
We reason the skeletal shape by minimizing the energy defined in Eq. 4. Our optimization variables include the vertex position v; at the canonical pose, the rigging weight w,, bone length by as well as the joint angles qf. All the variables except joint angles are shared across time, and joint angles are optimized per frame.
Initialization. We initialize the vertex position, bone length, and the rigging weight v,, by w: of canonical shape using the retrieved template character. The skeleton tree structure of this character is also taken as the basis of our skeletal parameterization. We initialize all the joint rotations as a unit quaternion.
The energy function is fully differentiable and can be optimized end-to-end. We use Adam optimizer [21] learn all the optimization variables. We adopt a scheduling strategy avoid getting stuck at a local minimum. Specifically, we first optimize the mesh scaling factor based on silhouette correspondences. We then jointly update the bone length scale, joint angles, and the neural offset field by minimizing the energy function defined in Eq. 4.
Optimization.
4 PlanetZoo dataset
Benchmarking 4D animal reconstruction requires large amount nonrigid action sequences with ground truth articulated 3D shapes. Towards this goal, construct synthetic dataset PlanetZoo

---

Table 1: Quantitative results on PlanetZoo.
Method
mloU f
mCham
Skinning +
Joint
Re-animation
ViSER [75]
0.190
0.133
6.398
0.416
0.028
LASR [74]
0.306
0.094
2.576
0.158
0.033
CASA (ours)
0.512
0.053
3.288
0.089
0.010
LASR
LASR
CASA (Ours)
CASA (Ours)
Input & Skeleton
Shape
Rigging
Input & Skeleton
Shape
Rigging
Figure 4: Reconstructed articulated shape. Top-left: input; top: LASR; bottom: ours.
consisting of hundreds of animated animals with textures and skeletons from different categories, Ap- pendix Fig. ?? depicts a snapshot of the assets and rendering images from our dataset, demonstrating the diversity and quality.
Data generation. We extracted animal meshes from the zoo simulator Planetzoo [35]. Cobra- tools [12] are used to extract those meshes along with their skeleton. With the extracted mesh and skeleton, we further render RGB maps, segmentation masks and optical flow for each frame using Blender.
Assets. To diverse environment, environmental lighting, and set floor textures with random materials from ambientCG². To reduce the gap between synthetic and real, we the location of the light source along with its strength to simulate different environments in real-world situations.
the background with random HDRI pictures for
Camera. To generate realistic action sequences, randomly change camera locations between every 12 frames, resulting in constant view-point change following the animal. The camera is allowed to rotate at certain angle, ranging from 15a to 22.5"
Articulation. In order to obtain animated animal sequences, for every 12 frames, 8 bones are selected from the skeleton tree with transformation attached. The angle value of rotation for each bone is sampled from uniform distribution. By doing this, we are able cover the whole action space, providing diverse action sequences.
Rendering, We generate silhouettes, optical flow, depth map, camera parameters, and RGB images using Vision Blender [7]. The physically based renderer is capable of showing fine details such as fur, making the rendering results more realistic. For each animal in the dataset, 180 frames are rendered.
Experiments
In this section, first introduce our experimental setting 5.1). We then compare CASA against comprehensive set of articulated baselines in various reconstruction and articulation metrics on both simulation ($ 5.2) and real-world datasets ($ 5.3). Finally, we demonstrate our inferred shape can be used for downstream reanimation tasks ($ 5.4).
5.1 Experimental setup
Benchmarks. We validate our proposed method on two datasets. Our proposed photorealistic rendering dataset PlanetZoo as well as real-world animal video dataset DAVIS[46), containing multiple real animal videos with mask annotations. For PlanetZoo, we choose 24 out of 249 total animals for testing and use the rest for validation and training. The testing dataset includes diverse articulated animals from unseen categories, including multiple quadrupeds, bipeds, birds categories,
hhttss//ammbeettg.com/

---

VISER
BANMo
LASR
CASA (Ours)
GT
Figure Qualitative comparison of our method on PlanetZoo.
VISER
BANMo
LASR
CASA (Ours)
Figure 6: Qualitative comparison of our method on real-world videos from DAVIS.
as well unseen articulated topology such as pinnipeds.
Metrics. We measure the reconstruction quality by Intersection over Union (IOU) and Chamfer distance, as well as skinning distance, joint distance and re-animation quality on PlanetZoo.
1) mean Intersection Over Union(mIOU) measures the volumetric similarity between two shapes. We voxelized the reference and the predicted shape into occupancy grids and calculate the IOU ratio.
2) mean Chamfer Distance(mCham) computes the bidirectional vertex-to-vertex distances between the reference and the predicted shape.
3) Joint is the symmetric Chamfer Distance between joints. We evaluate CD-J2J following [72]. Given predicted shape, we compute the Euclidean distance between each joint and its nearest joint in the reference shape, then divide it by the total joint number.
4) Skinning distance measures the similarity between the skinning weights. We first extract vertices associated with each joint using skinning weight. For each pair of joints from GT and prediction, we calculate the Chamfer distance between their associated vertices. Finally, we exploit the Jonker- Volgenant algorithm find the minimum distance matching between prediction and reference.
5) Re-animation measures how well we can re-pose an articulated shape to a target shape. Specifically, minimize the Chamfer Distance between the reference shape and the predicted shape by optimizing the joint angles of each bone for skeletal shape, or the rigid transform for control point- point-based shape. We consider this holistic metrie that jointly reflects the quality of shapes, skinning, and skeleton.
Baselines. We compare with state-of oftte----te-art approaches monocular-vided articulated shape reconstruction including LASR [74], ViSER [75], and BANMo [76].. Similar to our method, they utilize supervision of videos for training including segmentation masks and optical flows. We ground truth
download the open-source version of them from GitHub. For the input data, silhouctte either PlanetZoo dataset or DAVIS[46] dataset. We follow the optimization scripts in their code and get the baseline results.
3D reconstruction from a monocular video inevitably brings scale ambiguities. alleviate the issue of scale differences, for every predicted shape among all the competing baselines, we conduct a line
We thank Gengshan Yang for sharing the code and providing guidance in running the baseline algorithms.

---

LASR
CASA
GT
Reconstruction
Retarget@ +10
Retarget @++50
Retarget@
Figure 7: Reanimation results.
search to find the optimal scale that maximizes the IOU between the reference and the predicted shape.
5.2 Results on PlanetZoo
We present quantitative results PlanetZoo in Tab. 1. CASA achieves higher performance metrics mIOU and mCham, demonstrating a shape with higher fidelity. Our approach also outperforms other competing methods in reanimation, demonstrating the superior performance of holistic articulated shape reasoning and the potential for downstream reanimation tasks. CASA compares less favorably LASR in the skinning quality. We conjecture this is due additional degrees of freedom benefits brought by joint control point articulation.
Fig. 4 and Fig. 5 depict few qualitative examples of all competing algorithms PlanetZoo. We show the reconstructed mesh from both the camera view and the opposite view. The figure shows that CASA can recover accurate shape across various view angles under partial observation. In contrast, baselines fail to reconstruct reliable 3D mesh when the objects are partially observed. In addition, CASA also produces meshes with higher fidelity and local details in the visible region.
5.3 Real-world reconstruction
We demonstrate qualitative results of the competing algorithms on the real-world animal video dataset DAVIS [46] in Fig 6 and Fig. 4. This figure shows that both our method and baselines can get good reconstruction results from the camera perspective. But both two baselines fail to reconstruct those unseen parts. contrast, CASA can reliably reason the shape and articulation in the unseen regions, thanks to its symmetry constraints, skeletal parameterization, and 3D template retrieval.
5.4 Reanimation
now show how the reconstructed articulated 3D shape can retargeted to new poses. Given an inferred skeletal shape and the target GT shape at different articulated pose, apply inverse kinematic to compute the articulated pose to reanimate the inferred shape. Specifically, we optimize the articulated transforms such that the reanimated mesh is close to the target in Chamfer distance. Joint quaternions are optimized for skeletal mesh, and the articulation transforms are optimized for the control-poine ppoint-based method [74]. Fig. 7 shows comparison between the retargeted meshes of our methods and LASR on PlanetZoo, with GT target mesh as a reference. We observe that retargeted meshes look realistic and accurate and preserve geometric details. Despite having more degrees of freedom in reanimation, LASR fails to produce realistic retarget results. Tab. reports quantitative comparison in chamfer distance between the retargeted and the reference mesh, demonstrating our approach outperforms competing methods by a large margin.
Table Ablation study on energy terms*
Table 3: Ablation study on optimization settings.
Method
t
mCham I Skinning
Joint
Method
mIOU
mChum
Skinning
Joint
CASA
0.512
0.053
3.288
0.089
CASA
0.512
0.053
3,288
0.089
w/o Eass
0.011
wio offset
0.144
0.343
3,174
0.783
w/o Ex
0.427
0,350
3.290
0.636
wio disp. field
0.235
0.055
3.218
0.262
w/o Evm
0.387
4.008
wio skeleton
0.372
0.060
3,245
N/A
w/o Eureeh
0.449
0.041
3.228
0,330
w/o scaling
0.299
0.072
3.329
0.356

---

Table Ablation study on our retrieval strategy.
Table 5: Ablation study on initialization settings.
Method
mIOU
mCham
Skinning
Joint
Method
mIOU1
mCham
Skimning
Joint
CASA
0.512
0.053
3.288
0.089
CASA (retneval innt)
0.053
3.288
Retrieval-only (CLIP)
0,217
0.448
4.108
0.303
k-means rigging init
0.305
0.452
3.336
0.659
Retrieval (ImageNet)
0.972
Fixing skinning weight
0.052
3,235
5.5 Ablation study
We provide ablation study to demonstrate the efficacy of each design choice in our framework.
Energy terms. In Tab. 2, we ablate different terms of our energy function. Specifically, we consider the mask consistency, flow consistency, symmetry and smoothness regularization separately. We find that the mask part crucial for the performance, while the optical flow part also improves the framework results. Removing the symmetry offset term results in performance degradation since this term plays a vital role regularizing the neural offset, which can alleviate issues brought by the ambiguity of single-view monocular video. The smooth term leads to better qualitative results.
Optimization. We compare different optimization settings Tab. 3. The results show that: using neural offset field is superior to per-vertex offset or not deforming shape (CASA vs. w/o offset vs. w/o disp. field); stretchable skeleton helps (CASA vs. w/o skeleton); removing shape scaling step leads to performance degradation (CASA vs. w/o scaling).
Retrieval strategies. In Tab. 4, we test different retrieval strategies. Our result demonstrates that CLIP is the preferred backbone for retrieval, most likely due to training with significantly richer semantic information than ImageNet pretrained models.
Initialization. We test different initialization strategies for skinning weights in Tab. 5. We replace the rigging from the retrieved animal by using k- k-means for initializing weights. The comparison in the table shows that high-quality rigging weight initialization essential for good shape predictions. The results of sphere initialization also confirm the necessity of the proposed retrieval phase.
Stretchable bone. Fig. shows qualitative results with/without stretchable bone parameterization. Compared to merely deforming each vertex, stretchable bone deformation ensures global consistency and smoothness. As noted the figure, without stretchable bones, we see discontinuity the nose of the animal and a mismatch between the lower and upper mouth.
5.6 Limitations
While demonstrating superior performance in animal re- construction, our method has a few remaining drawbacks. Firstly, the source asset bank restricts the diversity of the retrieved articulation topology. Bone length optimization partially alleviates such limitations through "shrinking" bones. Still, it cannot add new bones to the kinematic tree, e.g., we cannot create a spider from quadrupedal template. Secondly, our method so far does not impose the constraint that bones are inside the mesh. We plan to tackle such challenges in the future.
stetchalee
Figure 8: Stretchable bone
6 Conclusion
In this paper, we propose CASA, a novel category-agnostic animation reconstruction algorithm. Our method take monocular RGB video and predict 4D skeletal mesh, including the surface, skeleton structure, skinning weight, and the joint angle at each frame. Our experimental results show that the proposed method achieves state-of-the-ar performances in challenging datasets. Importantly, we demonstrate that we could retarget our reconstructed 3D skeletal character and generate new animated sequences.
Acknowledgements. The authors thank Vlas Zyrianov and Albert Zhai for their feedback on the writing. The project is partially funded by the Illinois Smart Transportation Initiative STII-21-07. We also thank Nvidia for the Academic Hardware Grant.
See extremely large value
10

---

References
[1] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. In SIGGRAPH, 2005.
[2] Benjamin Biggs, Oliver Boyne, James Charles, Andrew Fitzgibbon, and Roberto Cipolla. Who left the dogs out? 3d animal reconstruction with expectation maximization in the loop. In ECCV, 2020.
[3] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Computer graphics and interactive techniques, 1999. 2
[4] Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael Black. Dynamic FAUST: Registering human bodies in motion. CVPR, 2017.
[5] Christoph Bregler, Aaron Hertzmann, and Henning Biermann. Recovering non-rigid 3d shape from image streams. In CVPR, 2000. 2
[6] Zhe Cao, Hang Gao, Karttikeya Mangalam, Qizhi Cai, Minh Vo, and Jitendra Malik. Long-term human motion prediction with scene context. In ECCV, 2020. 16
[7] João Cartucho, Samyakh Tukra, Yunpeng Li, Daniel S. Elson, and Stamatia Giannarou. Vision- blender: tool efficiently generate computer vision datasets for robotic surgery. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 2020. 2,5,
[8] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-rich 3d model repository. arXi::/5/2.03012 2015.
[9] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In CoRL, 2017. 3
[10] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoint without keypoints. In ECCV, 2020. 2
[11] Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Har- vey, Sergio Orts-Fscolano, Rohit Pandey, Jason Dourgarian, et al. The relightables: Volumetric performance capture of humans with realistic relighting. ACM TOG, 2019.
[12] HENDRIX-ZI -T22. cobra-tools. https: /tte::///itan /gtgttttttttt//ge//// comm/ppeniaaaa/cobreeeeeeoaoa tools, 2020. 7
[13] Yuan-Ting Hu, Jiahong Wang, Raymond A Yeh, and Alexander G Schwing. SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Data. In CVPR. 2021.
[14] Vuan-Ting Hu, Jiahong Wang, Raymond A Yeh, and Alexander G Schwing. Sail-vos 3d: A synthetic dataset and baselines for object detection and 3d mesh reconstruction from video data. In CVPR, 2021. 16
[15] Alec Jacobson and Olga Sorkine. Stretchable and twistable bones for skeletal shape deformation. In SIGGRAPH Asia, 2011. 3,4
[16] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
[17) Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timo- thy Scott Godisart, Bart Nabbe, lain Matthews. Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social interaction capture. PAMI, 2017.
[18) Angjoo Kanazawa, Michael J Black, David Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, 2018. 2

---

[19] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In ECCV, 2018. 2
[20] Ladislav Kavan, Steven Collins, Jifí Zára, and Carol O' O'Sullivan. Geometric skinning with approximate dual quaternion blending. ACM TOG, 2008. 3
[21] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 6
[22] Alexander Kirillov. Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. CVPR, 2020.
[23] Filippos Kokkinos and lasonas Kokkinos. Learning monocular 3d reconstruction of articulated categories from motion. In CVPR, 2021. 2
[24] Chen Kong and Simon Lucey. Deep non-rigid structure from motion. In ICCV, 2019. 2
[25] Nilesh Kulkarmi, Abhinav Gupta, David F Fouhey, and Shubham Tulsiani. Articulation- aware canonical surface mapping. In CVPR, 2020. 2
[26] Steven M LaValle. Planning algorithms. 2006.
[27] Jiahui Lei and Kostas Daniilidis. Cadex: Learning canonical deformation coordinate space for dynamic surface representation via neural homeomorphism. In CVPR, 2022. 2
[28] John P Lewis, Matt Cordner, and Nickson Fong. Pose space deformation: unified approach to shape interpolation and skeleton-driven deformation. In SIGGRAPH, 2000.
[29] Tianye Li, Timo Bolkart, Michael Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM TOG, 2017. 2
[30] Xueting Li, Sifei Liu, Shalini De Mello, Kihwan Kim, Xiaolong Wang, Ming- Hsuan Yang, and Jan Kautz. Online adaptation for consistent mesh reconstruction in the wild. In NeurIPS, 2020.
2
[31] Yang Li, Hikari Takehara, Takafumi, Taketomi, Bo Zheng, and Matthias NieBner. 4dcomplete: Non-rigid motion estimation beyond the observable surface. In ICCV, 2021. 2,3
[32] Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3d hand-object poses estimation with interactions in time. In CVPR, 2021. 2
[33] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In ICCV, 2019.
[34] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM TOG, 2015. 2,
[35] Ingo Lütkebohle. PlanetZoo Simulator. bttps://wwir http://wwwpaeeezoooa.eee.e.e com/, 2021. 2,
[36] Wei-Chiu Ma, Angi Joyce Yang, Shenlong Wang, Raquel Urtasun, and Antonio Torralba. Virtual correspondence: Humans as a cue for extreme-view geometry. In CVPR, 2022. 2
[37] Nadia Tlagnenat-Thalmmmn Richard Laperrire, and Daniel Thalmann. Joint-dependent local deformations for hand animation and object grasping. In Graphics interface, 1988. 3
[38] Bruce Merry, Patrick Marais, and James Gain. Animation space: A truly linear framework for character animation. ACM TOG, 2006.
[39] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In CVPR, 2022. 5
[40] Marko Mihajlovic, Yan Zhang, Michael J Black, and Siyu Tang. LEAP: Learning articulated occupancy of people. In CVPR, 2021.
[41] Richard A. Newcombe, Dieter Fox, and Steven M. Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In CVPR, 2015, 1,
12

---

[42] Michael Niemeyer, Lars Mescheder, Michael Oechsle. and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In ICCV, 2019.
[43] Atsuhiro Noguchi, Umar lqbal, Jonathan Tremblay, Tatsuya Harada, and Orazio Gallo. Watch it move: Unsupervised discovery of 3D joints for re-posing of articulated objects. In CVPR, 2022.
[44] Pablo Palafox, Aljaž Božič, Justus Thies, Matthias Nießner, and Angela Dai. Npms: Neural parametric models for 3d deformable shapes. In ICCV, 2021. 2
[45] Keunhong Park. Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo
Nerfies: Deformable neural radiance fields. In ICCV, 2021.
2
[46] F. Perazzi, J. Tont,Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine- Sorkine-Hornung, A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016.
2, 7, 8,
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 5,5
[48] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv:2204.06122 06/25, 2022.
[49] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang. Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guides dense prediction with context-aware prompting. In CVPR, 2022. 3
[50] Zhongzheng Ren, Xiaoming Zhao, and Alex Schwing. Class-agnostic reconstruction of dynamic objects from videos. In NeurlPS, 2021. 1,2
[51] Stephan R Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In ICCV, 2017.
[52] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In ICCV, 2021. 3
[53] Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. SIGGRAPH ASIA, 2017. 2
[54] Nadine Rüegg, Silvia Zuffi, Konrad Schindler, and Michael J Black. Barc: Learning to regress 3d dog shape from images by exploiting breed information. CVPR, 2022.
[55] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, 2019.
[56) Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd:
Multi-level pixel-
aligned implicit function for high-resolution 3d human digitization. In CVPR, 2020.
[57] Shunsuke Saito, Jinlong Yang. Qianli Ma, and Michael Black. Scanimate: Weakly supervised learning of skinned clothed avatar networks. In CVPR, 2021. 2
[58] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In ICCV, 2019.
[59] Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín-Martín, Linxi Fan, Guanzhi Wang, Claudia Pérez-D' Arpino, Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, et al. igibson 1.0: A simulation environment for interactive tasks in large realistic scenes. In IROS, 2020. 3
[60] Vikramjit Sidhu, Edgar Tretschk, Vladislav Golyanik, Antonio Agudo, and Christian Theobalt. Neural dense non-rigid structure from motion with latent space constraints. In ECCV, 2020. 2
13

---

[61]
Julian Straub, Thomas Whelan, Lingni Ma. Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge. Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv:1906.05797, 2019. 3
[62] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, 2019. 3
[63] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
[64] Shubham Tulsiani, Nilesh Kulkarni, and Abhinav Gupta. Implicit mesh reconstruction from unannotated image collections. arXiv:2007.08550 085044 2020. 2
[65] Yael Vinker, Ehsan Pajouheshgar. Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically-aware object sketching. SIGGRAPH, 2022. 3
[66] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and- image driven manipulation of neural radiance fields. In CVPR, 2022. 3,5
[67] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. Tartanair: A dataset to push the limits visual slam. In IROS, 2020. 3
[68] Xiaohuan Corina Wang and Cary Phillips. Weigi- enveloping: sqast-squares approxima- tion techniques for skin animation. In SIGGRAPH, 2002. 3
[69] Chung-Yi Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher- Shlizerman. HumanNeRF: Free-viewpoint rendering of moving people from monocular video. In CVPR, 2022.
[70] Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Dove: Learning deformable 3d objects by watching videos. arXiv:2107. 2021. 2
[71] Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding. In EMNLP, 2021. 3
[72] Zhan Xu, Yang Zhou, Evangelos Kalogerakis, Chris Landreth, and Karan Singh. Rignet: Neural rigging for articulated characters. ACM TOG, 2020. 8
[73] Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. NeurlPS, 32, 2019. 6
[74] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. LASR: Learning articulated shape reconstruc- tion from a monocular video. In CVPR, 2021. 2, 3, 4, 5, 6, 7, 8,9
[75] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan. Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. In NeurlPS, 2021. 2,4, 6, 7,8
[76] Gengshan Yang. Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In CVPR, 2022.
2, 3,
[77] Ze Yang, Siva Manivasagam, Ming Liang, Bin Yang, Wei-Chiu Ma, and Raquel Urtasun. Recovering and simulating pedestrians in the wild. In CoRL, 2020. 2
[78] Ze Yang, Shenlong Wang, Sivabalan Manivasagam, Zeng Huang, Wei-Chiu Ma, Xinchen Yan, Ersin Yumer, and Raquel Urtasun. S3: Neural shape, skeleton, and skinning fields for 3d human modeling. In CVPR, 2021. 2
14

---

[79] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. In CVPR, 2022.
[80] Xiaoming Zhao, Yuan-Ting Hu, Zhongzheng Ren, and Alexander G. Schwing. Occupancy Planes for Single-view RGB-D Human Reconstruction. In arXiv:2208.02887 2022. 2
[81] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. ECCV, 2020.
[82] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.
[83] Silvia Zuffi, Angjoo Kanazawa, Tanya Berger- Berger-Wolf, and Michael J Black. Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images In the Wild. In ICCV, 2019.
[84] Silvia Zuffi, Angjoo Kanazawa, and Michael Black. Lions and tigers and bears: Capturing non-rigid, 3d, articulated shape from images. In CVPR, 2018.
[85] Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black. 3d menagerie: Modeling the 3d shape and pose of animals. In CVPR, 2017. 2,
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [No] we will release the code and data upon acceptance
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] We eight GTX 2080 Ti and RTX A6000 GPUs.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [No] upon acceptance
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [Yes] BADJA is open-sourced; PlanetZoo has commercial license, following prior practices our community [51, 14], we will ask users to purchase the game before accessing the dataset.
(e) Did you discuss whether the data you are using/curating contains personally identi- fiable information or offensive content? [Yes] Both datasets do not contain personal identifiable information.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "VnZTuK6QUN", "reviewer": "Reviewer_ziQq", "summary": "This paper presents a category-agnostic character animation reconstrucion from a casual video input. To alliviate the ill-posed nature of the problem, this work first queries the closest template model from the database using CLIP features. As the following inverse graphics optimization stage can warm start with the retrieved template, the proposed approach produces better results both qualitatively and quantitatively over prior methods. Additionally, the paper introduces a large amount of synthetic dataset for qualitative evlauation of predicted attributes with diverse categories.", "questions": "- Please answer to the comments above.\n- It is not clear if CLIP is necessary for this retrieval task as text is not involved at all. It would be great if the use of various feature backbone including CLIP and ResNet pretrained with ImageNet is evaluated. \n- From the exposition, it is not clear how to compute CLIP feature from videos (CLIP only provides embedding per frame). Please elaborate.\n\nOther comments:\n- L43: The neural inverse graphics framework in general has been extensively used in prior works, and not novel. Please state more concretely what is novel over the prior works in terms of the optimization.\n- L231: Please add citation to Cobra-tools.", "limitations": "The limitation is discussed, but its societal impact is not.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper has the following strengths:\n- This work presents an interesting use of CLIP feature. To retrieve the closest animal template from the database, the CLIP features are extracted from both input video and synthetically rendered database. This type of semantic-based retrieval is more general and flexible than hand-crafted descriptors for the retreival task. \n- The paper presents an impressive qualitative results even from videos in the wild. \n- The proposed dataset would be a great resource for community for both training and evaluation to assess the accuracy of predicted attributes from diverse categories. \n\nThere are several weaknesses of this work:\n- The qualitative results of baseline methods (LASR, ViSER) are substantially worse than what’s presented in their original papers. I’m wondering if the results are cherrypicked or their code was not properly run. I would highly recommend reaching out to the authors of these papers to confirm that these results are expected. Please answer to this in the rebuttal to prove that the experiments are credible.\n- L58-60: I’m very confused about the notion of generalization here. The paper also discuss train/test split in L258. However, as far as I understand, this work presents an instance-specific training and there is no cross-instance generalization. Please clarify.\n\nOverall the paper presents an interesting approach and the results are impressive. However, the aforementioned concerns prevent me from giving a higher score at this stage. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VFydtWFE-cJ", "reviewer": "Reviewer_pF6x", "summary": "This paper focuses on the problem of 3D reconstruction of animals from a monocular video. The proposed method is claimed to be Category-agnostic, which means they can deal with different categories such as dogs, horses, but they are all quadruped. The major contributions is the category-agnostic reconstruction which is realized by first retrive a 3D template model from an asset. Then the template model is deformed and optimized to fit to the input video. ", "questions": "I have some questions or confuse on the some technical details of the proposed method, \n1) How to optimize both the bone length, joints angles, skinning weights together? Do we need to optimize one while fixing others? Furthermore, what is the improvement of optimizing skinning weights, what if the skinning weights are not optimized, but using those from retrived template?\n\n2) How is the optimization initialzed? How can we achieve the initial fitting to images or videos, including initial global pose?\n\nThe authors might also want to include the response to the issues I raised in Weakness above.", "limitations": "The major limitation is lack of technical contribution. Using the pre-trained CLIP model as features to retrive the 3D template is good, but I'm not sure this could be claimed as technical novelty. And the following optimization is also pretty standard. The authors should point out what are the major technical contributions that really stands out.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nFirst, having a good initial template to start with the following up optimization will certainly reduce the deformation space and make the optimization to be more feasible. They have demonstrated better visualization results by adopting this retrived template model. The idea is pretty easy to understand and the paper is organized and written well.\n\nWeakness:\n1) Obtaining this template model by retrival using the pre-trained CLIP model is more or less an engineering work. I don't think this could be claimed as a technical contribution. The performance is improved mainly due to this retrived template, while the compared approaches start with some general shape for example, a sphere.\n2) The optimization pipeline or the loss function is not novel. It is pretty much a standard optimization, I'm a bit confused what the authors want to claim on this optimization problem. In addition, the skeletal representation of articulated models, they are just standard way of dealing with those kinds of animals. I'm not sure what the authors want to emphasize on this.\n3) Missing important comparison:  -- BANMo: Building Animatable 3D Neural Models from Many Casual Videos. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6XzKgobF_FU", "reviewer": "Reviewer_4EwH", "summary": "The authors propose CASA, a method for recovering 3D shape and skeletal movements from monocular videos. Given a video, it first retrieves a shape and a skeleton from a library with 200+ animals, then optimizes both shape and articulations with differentiable rendering. Results are shown synthetic real datasets, with applications in re-posing. A new dataset PlanetZoo with 200+ animated animals is introduced.", "questions": "- How many categories does the dataset contain? What are the categories?\n- How are the root node transformations represented and initialized? Are they the same as LASR/ViSER?\n- During model retrieval, how to sample viewpoints when rendering the 3D model? How many viewpoints are needed? How long does it take to sampling and matching? Also, pairwise matching should produce SxNxT scores, where S is the number of shapes, and N is the number of renderings per shape, and  T is the number of frames. Then how is T and N marginalized?\n", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The proposed PlanetZoo dataset is interesting, as I'm not aware of a dataset of similar size (>200 animals) and quality (with texture, skeleton and deformation). It has potential to be used to evaluate and benchmark the performance of animal 3D reconstruction algorithms. From that perspective, it would be helpful to highlight the features of the dataset, and compare with the existing ones (such as deforming things 4D).\n- The method is sensible. It leverages 3D shape and skeletal priors of specific categories to improve the reconstruction quality. Since the library is large, the method can be applied to a wide range of animal categories.\n- The usage of skeleton structure also helps the re-posing application and user controllability.\n\n**Weakness**\n- Quality of results. The results are not faithful to the input data even test-time optimization is used. For instance in Fig.7, the reference cow image does not contain a horn but the reconstruction has horns, possibly from the template being retrieved. This is not desirable. I'm also confused as I did not notice a term that prevents the template shape to change when there is disagreement with the image evidence during optimization. This makes it unclear whether the optimization works properly.\n- The experimental comparison can be made fairer. LASR/ViSER does not have access to a 3D shape library. Proper baselines would be methods that use a 3D shape template, such as ACSM [A], ACFM, or proving LASR a 3D template.\n- There are some missing details (see questions). The one I'm most concerned about is the initialization of root transformations, which is crucial for reducing the ambiguity between shape and deformation. For instance, if the heading direction is rotated by 180 deg at initialization (the head becomes the tail), the optimization might focus on deforming the shape without correcting the heading.\n- Ablation study is missing in the main paper. How well does retrieval perform? As it is a major contribution, a quantitative evaluation and thorough analysis is expected. For the ablation study on flexible bone model, a qualitative comparison is desired as numbers does not give much insight in this context.\n- Fig. 3 is slightly misleading as it is not clear which parameters are per-frame and which are per-video. The subtitles also conflated per-video deformation (stretch and deform) vs per-frame deformation (rigging), vertex deformation (deform) vs bone deformation (stretch). \n\n**Other related work**\n- [A] Articulation-aware canonical surface mapping. CVPR 20.\n- [B] Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects, CVPR 22.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept