Learning Implicit Representation for Reconstructing Articulated Objects
Authors: Hao Zhang
Fang Li
Samyak Rawlekar
Narendra Ahuja
Venue: ICLR 2024 poster (2024)
Keywords: 3D reconstruction from videos
Articulated Objects
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: 3D Reconstruction of moving articulated objects without additional information about object structure is a challenging problem. Current methods overcome such challenges by employing category-specific skeletal models. Consequently, they do not generalize well to articulated objects in the wild. We treat an articulated object as an unknown, semi-rigid skeletal structure surrounded by nonrigid material (e.g., skin). Our method simultaneously estimates the visible (explicit) representation (3D shapes, colors, camera parameters) and the underlying (implicit) skeletal representation, from motion cues in the object video without 3D supervision. Our implicit representation consists of four parts. (1) skeleton, which specifies which semi-rigid parts are connected. (2) Semi-rigid Part Assignment, which associates each surface vertex with a semi-rigid part. (3) Rigidity Coefficients, specifying the articulation of the local surface. (4) Time-Varying Transformations, which specify the skeletal motion and surface deformation parameters. We introduce an algorithm that uses these constraints as regularization terms and iteratively estimates both implicit and explicit representations. Our method is category-agnostic, thus eliminating the need for category-specific skeletons, we show that our method outperforms state-of-the-art across standard video datasets.

Published as a conference paper at ICLR 2024
LEARNING IMPLICIT REPRESENTATION FOR RECON- STRUCTING ARTICULATED OBJECTS
Hao Zhang, Fang Li, Samyak Rawlekar & Narendra Ahuja Department of Electrical and Computer Engineering University of Illinois Urbana-Champaign (hacz19, fangli3, samyakr2, n-ahu s
ABSTRACT
3D Reconstruction of moving articulated objects without additional information about object structure is a challenging problem. Current methods overcome such challenges by employing category- sppecifcc skeletal models. Consequently, they do not generalize well to articulated objects in the wild. We treat an articulated object as an unknown, semi-rigid skeletal structure surrounded by nonrigid material (e.g., skin). Our method simultaneously estimates the visible (explicit) representation (3D shapes, colors, camera parameters) and the implicit skeletal representation, from motion cues in the object video without 3D supervision. Our implicit repre- sentation consists of four parts. (1) Skeleton, which specifies how semi-rigid parts are connected. (2) Skinning Weights, which associates each surface vertex with semi-rigid parts with probability. (3) Rigidity Coefficients, specifying the articu- lation of the local surface. (4) Time-Varying Transformations, which specify the skeletal motion and surface deformation parameters. We introduce an algorithm that uses physical constraints as regularization terms and iteratively estimates both implicit and explicit representations. Our method is category-agnostic, thus elim- inating the need for category-spent -specffc skeletons, we show that our method out- performs state-of-the-an across standard video datasets. The code is available on GitHub at: http:///githuu.coo/aoooooo wwt.haooooo/LLMM IMLL
INTRODUCTION
monocular videos input, goal is to reconstruct the 3D shape of the object in motion within these videos. The object in the video articulated and exhibits two distinct types of movement: (1) Skeletal motion, which arises from the movement of its articulated bones, and (2) Surface deformation, which arises from the movement of the object's surface, such as human skin. Skeleton shape easily represented by bone parameters, joints, and skeletal motion by their movements. Surface shape is usually represented by an irregular grid of 3D vertices placed along the surface, and the deformation by the movement of the vertices.
Given
Certain methods Pumarola et al. (2021) learn vertex deformations over time without differentiating the two distinct movement types, resulting in high computational costs and non- non-smooth transitions in motion estimates. Subsequent methods Yang et al. (2021a] b y:22): Lewis et al. (2023); y Kulkarni al. (2020) do separate the two motions, yielding better computational efficiency and smoother motion estimates. They use the Blend Skinning technique, whose efficiency hinges on the initial positions of the bones. They attempt to select better initial estimates by associating bones with groups of nearby vertices, estimated by applying K-means K-means-clustering clusterrig the mesh vertices. Not being physically valid, this representation still yields non- con-intuitive bone estimates. Methods like Hum Weng al. (2022) and RAC Yang al. (2023) morphological information to boost the 3D reconstruction using category-specife skeletons and pose models, which restricts the ability autonomously acquire structural knowledge from motion.
To address the above issues, we introduce LIMR (Learning implicit Representation). method to model not only the visible (explicit) surface shape, and color, but also the implicit sources, by
'The support of the Office of Naval Research under grant 1-104400---24444 and of USDA National Insti- tute of Food and Agriculture under grant 2020-67021-32799//002444 are gratefully acknowledged.

---

Published as a conference paper at ICLR 2024
Canon g
(Sc. 3.g
Reade Leset
BuD emage, Nato
Fanaut Skiison (Fw. 31)
Rackwar Blra sceang
0
Camana
Opial Flw
diee M.D
Undrr (Ses 3.3
Updir Skeke
Boes Combbrcna
Meken Conalrints
Opecal Flw fast Oge
Optori Floe
Bir Birri
Wary
Dxplict Heproamarse
Bacionad Ban
vagi
(Soe 3.))
Figure 1: Method Overview. LIMR optimizes both the explicit representations Re: e.g., surface mesh, color M. and camera parameters Pc, implicit representation R4, e.g., skeleton ST. skinning weights W, rigidity coefficients R derived from W, and time-varying transformation T, for root body and semi-rigid parts in time t, in an iterative manner. We optimize R. using differentiable rendering frameworks (3.2 {), and optimize R; using physical constraints (3.1
We optimize
R. and R. using the SIOS algorithm 3.3
estimating the skeleton, its semi-rigid parts, their motions, and the articulated motion parameters given by rigidity coefficients. Our method and results show that the information present the video cues allows such decomposition.
LIMR models the above two types of motion by learning two corresponding representations: ex- plicit and implicit. Similar to the EM algorithm we introduce the Synergistic Iterative Optimization of Shape and Skeleton (SIOS²) algorithm to learn both representations iteratively. During the Ex- pectation (E) phase, we update the implicit skeleton by employing our current explicit reconstruction model to calculate the 2D motion direction for each semi-rigid part (bone), as well as measure the distances between connected joints at the ends of bone. Then update the skeleton using two physical constraints: (1) The direction the optical flow should be similar within each semi-rigid part, and (2) distances between connected joints (bone length) should remain constant across frames. Following this, in the Maximization (M) phase, we optimize our 3D model using the updated skele- ton.
The main contributions this paper are as follows: (1) To the best of our knowledge, LIMR the first learn implicit representation from one (or more) RGB videos and leverage it for improving 3D reconstruction. (2) This shape improvement is achieved by obtaining a skeleton that consists of bone-like structures like physical skeleton (although not truly one), skinning weights, and rigidity coefficients. (3) Along with the implicit representation, our SIOS? algorithm also synergis- tically optimizes the explicit representation. (4) Because LIMR derives its estimates without any prior knowledge of object shape, it is category aggnssiic.... (5) Experiments with a number of stan- dard videos show that LIMR improves 3D reconstruction performance with respect to 2D Keypoint Transfer Accuracy and 3D Chamfer Distance in the range of 3%-8.3% and 7.9%-1 144 over state-of- the-art methods.
2 RELATED WORK
Template-Guided and Class-Specific Reconstructions. Various 3D template methods em- ployed differently: some deform template vertices Zuffi et al. (2018), others segment the mesh, operate on these parts, and reassemble for reconstruction Zuffi al. (2017): Xiang al. (2019). Some use efficient vertex deformation through blend skimning Wang & Phillips (2002); Kavan et al. (2007). Some introduce 3D poses, learning joint angles with ground truth data for better recon- struction Zuffi et al. (2019): [Biggs et al. (2020); BBdgger et al. (2020); Kocabas et al. (2020). These approaches excel with known object categories and abundant 3D data but struggle with limited 3D data unknown categories. RAC Yang et al. (2023) recently introduced category- speciffc shape models, yielding good 3D reconstruction. Current trends aim to minimize reliance on 3D annota- tions, opting for 2D annotations like silhouettes and key points Goel et al. (020);; Kanazawa et al. (2018); Li et al. (2020b). Single-view image reconstruction achieves impressive results without ex- plicit 3D annotations Li al. (2020a): Kanazawa et al.) 2018): Kulkarni et al. (202)) MagicPony
2

---

Published as a conference paper at ICLR 2024
Wu et al. (203b) learns 3D models for objects such as horses and birds from single-view images, but faces challenges with fine details and substantial deformations, especially for unfamiliar objects.
Class-Independeni and Template free methods for Video-Based Reconstruction. Nonrigid structure from motion (NRSIM) does not rely on 3D data or annotations. It utilizes off-the-shelf 2D key points and optical flow for videos, regardless of the category [Tedd & Deng (2020). Despite handling generic shapes, NRSIM requires consistent long-term point tracking, which isn't always available. Recently neural networks have learned 3D structural details from 2D annotations for specific categories. However, they struggle with long-range and rapid motion, particularly in uncon- trolled video settings. LASR and ViSER Yang et al. (2021a reconstruct articulated shapes from monocular videos using differentiable renderingLiu et al. (2019). albeit sometimes yielding blurry geometry and unrealistic articulations. Banmo Yang et al. (2022) addresses this by leveraging nu- merous frames from multiple videos to generate more plausible results. However, obtaining such a diverse collection of videos with varying viewpoints can be challenging.
Neural Radiance Fields. In scenarios with registered camera poses, NeRF and its variations Wang jtt al. (22Ibb); Jeong et al. (202T); ); Lin et al. ); Wang et al. (2021b); Li et al. (2023) typically optimize a continuous volumetric scene function to synthesize novel views within static scenes con- taining rigid objects. Some approaches Pumarola et al. (202T): ; Li al. (202T) Park et al. (202 b);
Tretschk et al. (202I) attempt to handle dynamic scenes by introducing new functions to transform time-varying points into a canonical space. However, they struggle when confronted with signif- icant relative motion from the background. To address these challenges, pose-controllable NeRFs Wu et al. (0022): Liu et al. (2021) have been proposed, but they either heavily rely on predefined category-level 3D data or synchronized multi-view videos. In our method, refrain from using any provided ground truth 3D information during optimization and can generate more 3D information compared to other techniques.
3
METHOD
Given one or more videos of articulated objects, our method learns explicit representations R. (M, well as implicit representation R, (Sr, W. R, T) synergistically. M represents the surface mesh and color in the object-centered (canonical) space. P are camera parameters in frame t. The implicit representation includes the skeleton ST {B E _Bx13 E RR ). the skinning weights W E IR N* B transformations of root body and semi-rigid parts T (Th T B. T E ESE(3). Bones B include the Gaussian centers (C = B 3), precision matrices (Q = B 1. 12)), and lengths of bones (B :, 12). J denotes the joints, containing the indices of two connected bones (J(::: 2) and the coordinates of the joint (J 2 :)). B, F,, and J are the numbers of bones, edges between vertices, and joints. The implicit representations are optimized using physical constraints such as bone length being consistent and optical flow directions being similar in the same semi-rigid parts across time (Sec. (.1) The implicit representations are optimized using physical constraints such as bone length being consistent and optical flow directions being similar in the same semi-rigid parts across time (Sec. [.J). and the explicit representations are optimized using differentiable rendering (Sec. 3.2 We leverage the Synergistic Iterative Optimization of Shape and Skeleton (SIOS?) algorithm to optimize both representations in an iterative manner (Sec. 3.3
Rxxss,
the vertices-wise rigid coefficients R E F and the time-varying
3.1 IMPLICIT REPRESENTATION LEARNING
Skeleton Initialization by Mesh Contraction. Given a mesh M- = (X, E). where X, E are ver- tices and edges. instead of using K-means to cluster centers from vertices, we use a Laplacian Contraction Cao et al.) (2010) to obtain the initial skeleton ST. It starts by contracting the mesh geometry into zero-vor vero-voluue skeletal shape using implicit Laplacian smoothing with global posi- tional constraints; details are given in A. This iterative contraction captures essential features, usually in fewer than ten iterations, with convergence when the mesh volume approaches zero. This contraction process retains key features of the original mesh and does not alter connectivity. The contracted mesh is then converted into a 1D skeleton through the connectivity surgery process of et al. (2008) that removes all collapsed faces while preserving the shape and topology of the contracted mesh. In the process of connectivity surgery on a contracted 2D mesh edge collapses driven by dual-component cost function: shape term and sampling term. The shape cost,
3

---

Published as a conference paper at ICLR 2024
2D Camera Space
3D Cartera Space
Skeleton
Badkyart Warp
Ireese Staal
Stnnnag
2D Flow Dinaction (Ml) (IXWK22
Surfhce Flow Direction 1"
Vishiliy Marri V: x
Port Decompoettion x
Bote Metion Directiot p*
Figure 2: Optical Flow Warp. We backward project the 2D optical flow to the camera space, obtaining flow direction FS for every vertex the surface and calculate the visibility matrix V according to the viewpoint. Then we apply inverse blend skinning with F2D, V and skinning weights W as inputs calculate the bone motion direction F Note t denotes mapping from frame to t 1.
inspired by the QEM simplification method Garland & Heckbert (1997). aims to retain the inherent geometry of the original mesh during simplification, ensuring minimal shape distortion. Conversely, the sampling cost is tailored to deter the formation of disproportionately long edges by assessing the cumulative distance traveled by adjacent edges during an edge collapse. This consideration prevents over-simplification in straight mesh regions, preserving granular and true representation of the mesh's original structure.
Skinning Weights. For given vertices the surface, define soft skinning weights matrix W E R B which assigns N vertices to B bones probabilistically. Optimization may be difficult for learning such matrix. Therefore, similar to BANMo, obtain the decomposition matrix by calculating the Mahalanobis distance between surface vertices and B bones as follows:
W n. b soflmax(d((X)))mm C₁ Qn) MLP (X.)); d(X,, Cb Q+) (X, Cb) Cb), Q = V A,
(1)
where d is the distance function between vertex and bone b. Each bone is defined as a Gaussian ellipsoid, which contains three learnable parameters: center C e R² x 3, orientation V R and diagonal scale A IR. Bx333 and X,, denotes the coordinates of vertex n.
BBx333
Rigidity Coefficient. significant factor in representing deformations within an articulated object the degrees of freedom each vertex possesses relative to its neighboring vertices on the surface. To systematically account for this, we introduce a rigidity coefficient matrix, R, with dimensions E where E denotes the total edge count linking the vertices. It is important recognize that regions around joint areas are inherently more pliable and exhibit pronounced deformations. This is in stark contrast to vertices anchored in the midst areas of the semi-rigid components. Such differential susceptibility leads to observable correlated movements amongst proximate vertices. For given skeleton, represented by ST, and the corresponding semi-rigid decomposition W of the object, the coefficient matrix R. is formulated by computing the product of entropies from the probability distributions of two connected vertices assigned to bones:
Σ W. W. log, W.b +1) b=0
Σ W W logz log, W5. + +2)
R
(2)
Here, Ri,y signifies the rigidity coefficient connecting vertices i and j. The term acts stabi- lization constant avoid division by zero, with default setting at 0.1. We restrict our computations to pairs of directly linked vertices.
Dynamic Rigid. Capitalizing on this rigidity coefficient, we propose the DR (Dynamic Rigid) loss, serving as an improvement over the conventional ARAP (As Rigid As Possible) loss, which has been extensively adopted as a motion constraint in several prior works Yang al. (202 la); Sumner et al. (2007); Tulsiani et al. (2020); Yang et al. (2021b). ARAP loss encourages the distance between all adjacent vertices to remain constant across continuing frames. However, this approach is not always helpful. Specifically, vertices near the joints should inherently possess more degrees of freedom compared to those located the midst of semi-rigid parts. To address this, our proposed Dynamic Rigidity (DR) offers significant enhancement, allowing for natural and adaptive

---

Published as a conference paper at ICLR 2024
spatial flexibility based on the vertex's proximity joints:
LDR Σ
R
X
x1,12 -|xx--1 -x--1
(3)
where X denotes the coordinates of vertex i at time & and N; represents the set of neighboring vertices of vertex i.
Blend Skinning. We utilize blend skinning map the surface vertex X from canonical space (time 0) to camera space time t: X Given bones B, skinning weights W and time-varying transforma- tion T {T have: X T F3 b= W n.b N
where each vertex is transformed by combining the weighted bone transformation Th b > O and then transformed to the camera space by root body transformation To On the contrary, we employ the backward blending skinning oper- ation to map vertices from the camera space to the canonical space. The sole distinction from blend skinning lies in utilizing the inverse the transformation instead of T.
Optical Flow Warp. Given the 2D optical flow between times t and t + 1, represented as F our goal is to determine the 2D motion direction for each semi-rigid component (underlying bone). This bone motion direction is denoted as F and is expressed in R Bx2 Notably, these directions serve as a critical physical constraint in the skeleton refinement process, as outlined in Section 3.3 Intuitively, the motion of bone is aggregate of the motions of the vertices associated with it. To achieve this, we project each pixel from the 2D optical flow back to the 3D camera space. Due to the fact that vertices may map perfectly onto pixels, the surface flow direction FS. E RNNx x2 is derived using bilinear interpolation. It is essential to note that attributing optical flow direction to vertices on the non- visible side of the surface erroneous. To address this, we compute visibility matrix D E RN based on the current mesh and viewpoint of time t using ray-casting. Optical flow directions corresponding to non-visible vertices are then nullified. Then, we approach the estimation of bone motion direction by employing blend skinning, which coherently integrates the optical flow directions associated with surface vertices. Specifically, given a defined skinning weights, represented as W, and the computed surface flow direction FSA, the bone motion direction can be represented as:
F
Σ W WniFSSV
FB. (F ..... FBL
(4)
In the above equation, FB signifies the flow direction associated with bone 6, while FS. and V denote the flow direction and visibility status for vertex n, respectively.
Joint Localization & Bone Length Calculation. Given the current bone coordinates B, bone connections J[:.: 2). and the skinning weights W, we first identify the set of vertices Va, that lie above the joint Jı,j connecting bones and j. vertex v. included in Vi, both W.,; and W are greater than or equal to 4,- (default by 0.4). The coordinates of Ji.j are then computed
as the mean coordinates of the vertices Vi,- Specifically, [t, :] Ev,cy, X.
[.,ll where X,,
represents the coordinates of Un- Once we have the coordinates for each joint, the length of each bone is calculated the distance between the coordinates of the corresponding joint pairs.
3.2 EXPLICIT REPRESENTATION MODEL
There have been two main approaches to learning explicit representations (R.): NeRF-based ap- proach Waan et (2021a) a Pumarola et al. (202I), )), and the neural mesh renderer as described in soft rasterizer Liu et al. (2019)
Losses Regularizations. Similar to BANMo, the Nerf-based approaches use reconstruction loss, feature loss, and regularization term: LNeRF bused Ereconstruction + Lieature + L consistency The recon- struction loss follows the existing differentiable rendering pipelines Yariv et al. (2020); Mildenhall et al. (2021), which is the sum of Lalloouetee LRGB. and Loptical-Foow- The feature loss is composed of the 3D feature embedding loss (Creaure-embeddnng) by minimizing the difference between the canonical embeddings from prediction and backward warping. The 3D consistency loss (C3D-comiitency) from NSFF et al. (22[)) 1 is used to ensure the narward-defomee 3D points match their original location upon hackward-deformation
A.2
5

---

Published as a conference paper at ICLR 2024
DANIS's
HaretZoe's Dance
AMA'S
(CLASK
(4) BANMe
(c) LASK
(40 BANMo
LALA
Figure 3: Mesh Results. We show the reconstruction results of (a) Our approach, (b) Our approach w/o part refinement, LASR, and (d) BANMo in the DAVIS's camel, dance-twir and Plan- etZoo's zebra.
The loss function of Neural Mesh Rasterization- methods includes reconstruction loss, mo- tion, and shape regularization: LNMR based Reecomstraction + percepiual a and are set the same as LASR. The reconstruction loss is the same as the NeRF-based, with the addition of perceptual loss Zhang et al. (2018). Furthermore, we shape and motion regulariza- tions, and the soft-symmetry constraints. We replace ARAP loss with Dynamic Rigid (DR) loss to encourage the distance between two connected vertices, within the same semi-rigid part, to remain constant. The Laplacian operator is applied to generate a smooth surface (Cshape). The details are presented in Appendix A.2
x Lahage + 17 x LDR where
Part Refinement. For the NMR-based scheme, the silhouette loss contributes the most to the ren- dering loss. For objects like quadrupeds whose thin limbs only occupy small part of the 2D mask, the contribution of limbs to the overall silhouette loss is relatively less. This phenomenon always results in well-reconstructed torso, but bad-reconstructed limbs. So with the skinning weights W E R N/ B obtained in S we then train only the limb parts by freezing all parameters but those for limbs. The selection process is given by W Wone-hot i, W[W[i.] :] = argmax; W[I, x
and W one E {0. JN represent the one-hot encoded decomposition matrix, where each vertex assigned exactly one part. Given W hote we update the vertices only from small parts.
3.3 SYNERGISTIC ITERATIVE OPTIMIZATION
Our approach is grounded two interdependent sets of learnable parameters. The first set is sociated with the reconstruction model, comprising the mesh, color, camera parameters, and time- varying transformations. Simultaneously, the secondary pertains to the underlying skeleton, encompassing the bones and articulation joints. To achieve a harmonized optimization of these parameter sets, we utilize the Synergistic Iterative Optimization of Shape and Skeleton (SIOS?) al- gorithm, as elucidated in Alg T This algorithm updates both parameter sets iteratively similar to the EM algorithm. During the E-step, the skeleton remains fixed and is utilized in the blend skinning operation and regularization losses. The reconstruction model is then updated based on the render- ing losses and regularization components. Subsequently, using the updated reconstruction model, we determine the bone motion direction and bone lengths for the selected frames. This information is utilized in refining the skeleton in light of physical constraints.
Skeleton Refinement. In the M-step, the skeleton is updated using two constraints: (1) Merge bones B,, By if they consistently move in syne across H selected images {I, / indicating they belong to a single semi-rigid part. (2) Introduce a joint between Jy if their distances fluctuate significantly across specific frames. Given that,
if max Length (B/) min Length(B/) ta
introduced;
if min S(F Fh >t
By and BB" merged to form B,
J B
where S(a, b) x bla is the cosine similarity. Furthermore, the coordinate of J is computed by the mean of J, and J and we remove the connection between them and connect them J. B exp Eeaw W..b
b
w/. By whr Byc, with w/ exp Eb-t W W.. texp Eo- W.. Wn.s which is applied on Gaussian centers

---

Published as a conference paper at ICLR 2024
DAMSS's
DANS'S
AMAS wwW.rrhhammoog
DAWN'SS
Video
All vodoo
25
Soload FAbOs
254
Skeleton Topelogy Samieiigd Decomposttion different (o) Eo values
(b) mumber of videos
(c) RigNet
Figure 4: Implicit Representation Results: (a) variations with different to values. (b) from different videos. From left to right are the results for DAVIS's camel, sonnce-twirll AMA's BANMo's human-cap. (c) shows the skeleton generated by RigNet Xu et (2000) B indicates the number of bones.
AMA'ss swing, and
C' and precision matrices Q but the new bone length is calculated directly from the coordinates of the joints at the far end of two bones.
4 EXPERIMENTAL RESULTS
In this section, evaluate our approach with variety of experiments. The experimental evalua- tion is divided into two scenarios: 3D reconstruction leveraging (1) single monocular short videos, and (2) videos spanning multiple perspectives. We select the state-of-the-ar methods LASR Yang itt al. a2) and BANMo Yang et al. (2022) as our baselines for the two respective scenarios men- tioned above. For fair comparison,
adopt NeRF-Based and NMR-Based explicit representation models when compared with BANMo and LASR respectively. Additional details and results for the benchmarks and implementation can be found the supplementary material, and we plan to make the source code publicly available.
4.1 RECONSTRUCTION FROM SINGLE VIDEO
Compared to using multiple videos, reconstructing from a single short monocular video evidently more challenging. Hence, we aim to verify if our proposed implicit representation can achieve robust representation under such data-limited conditions. Firstly, we tested our approach on the well-established BADJA BBiggs et al. (2019) benchmark derived from DAVIS Perazzi et al.] (2016) dataset. Additionally, we broadened our experimental scope by introducing the PlanetZoo dataset manually collected from YouTube. The Plantzoo dataset amplifies more notable challenge than the DAVIS dataset on the more extensive camera motion of each video contributing to the biggest distinction. Datasets details provided in Sec[A.4]
Qualitative Comparisons. We present a comparative analysis of mesh reconstruction with LIMR, LASR, and BANMo, illustrated in Fig 3 With single monocular video, e.g. DAVIS's camel, dance-twi 1 1 and PlanetZoo's zebra, BANMo does not generate satisfactory results as depicted in (d) due the constraints of NeRF-based explicit representation models, demanding massive input video and adequate viewpoints. Conversely, NMR-based methodologies like LASR exhibit acceptable results. However, the absence of an understanding of articulated configurations of objects induces notable inaccuracies. For instance in Fig camel, LASR erroneously elongates vertices beneath the abdominal region and conflates the posterior extremities align the rendered mask with the ground truth mask, thus falling into an incorrect local optimum. LIMR, as shown in Fig (a), learns the implicit representation which helps it overcome problems like those seen with LASR. Our approach steers the canonical mesh with the Iearned skeleton, ensuring alignment with precise skeletal dynamics and satisfactory reconstruction outcomes. LIMR is lenient with regard to video and view requisites. It learns a skeleton from even single monocular video.
Quantitative Comparisons. Table T shows the performance of LIMR concerning 2D key points transfer accuracy against state-of-the-as art methods: LASR and ViSER Yang al. (2021b), span- ning both DAVIS and PlanetZoo datasets. While LASR and ViSER have their respective strengths depending the videos, LIMR consistently outperforms both of them across all tested videos.
7

---

Published as a conference paper at ICLR 2024
Table 2D Keypoint transfer accuracy on DAVIS and PlanetZoo videos.
DAVIS
PlanetZoo elephant
Method
camel
dog
cow
bear
dance
Ave.
dog
zebra
bear
dinosaur
Ave.
VISTER
76.7
65.1
77.6
72.7
78.3
74.1
LASR
78.3
60.3
82.5
83.1
55.3
71.9
73.4
57.4
69.5
63.1
71.3
66.9
Ours wil DR
79.1
65.4
K3.2
85.3
75.9
78.5
74.7
58.3
70.1
64.9
12.8
68.2
Ours
80.3
67.1
83.4
86.8
83.1
80.2
77.5
61.1
70.9
66.6
73.6
69.9
free
Swar
LASK
LAWE
Figure 5: Rendering Results. Compare the rendering results on DAVIS's camel, dance-twirl with prior art LASR.
Specifically, within the DAVIS dataset, our approach surpasses LASR by 8.3% and ViSER by 6.1% and achieves 3% advantage over L.SRR the Planet/lo dataset. Note that ViSER is sensitive large camera movement, and a large number input frames, which causes ViSER to perform badly in the PlanetZoo. Given BANMo's inability to produce commendable outcomes from a single short monocular video, we did not compare LIMR with BANMo.
DANSSS
DAVE'S dance
humencca
Figure 6: Bone localization and part decomposition results from LASR.
G1
Figure 7: Initial skeleton obtained via mesh contraction.
4.2 RECONSTRUCTION FROM MULTIPLE VIDEOS
For evaluating the reconstruction with multiple video inputs, we conducted experiments BANMo's cat, human camaa AMA's swing, and samba datasets. As illustrated in Fig3swirrs (b), compared with BANMo, our method further refines skeletal motion through the assimilation of implicit representations. Through observation, compared to BANMo, our method can achieve more accurate reconstruction results in
with larger motion ranges, such the knees and elbows. Notably, BANMo requires a pre-defined number of bones, which is defaulted by 25. In contrast, we achieved better
learn the most suitable number of bones adaptively. In the swi experiment, results than BANMo using only 19 bones rather than 25. Also, quantitative results of LIMR on AMA's Swing and Samba compared with BANMo are shown in Tab Under equivalent training conditions, our method employs fewer bones yet outperforms BANMo by around 12%.
4.3 DIAGNOSTICS
Here we ablate the importance of each component, show the outcomes with different training set- tings, and analyze why our method outperforms the existing methods.
Physical-Like Skeleton vs Virtual Bones. The key distinction of our approach from existing meth- ods our pursuit learn skeleton for articulated objects, contrast to the commonly virtual bones. As demonstrated in Fig 6, LASR tends to concentrate bones in the torso area, allocating only few bones the limbs. However, in practice, limbs often undergo more significant skeletal motion compared to the torso. This discrepancy causes existing methods to underperform, particularly at limb joints. In 6 the entire limb of the camel has just one bone, which is treated as a semi- rigid part, and prevents bending. Rendering results in Fig 5 3 from the dance-twl wirl, camel, dog, and zebra experiments show our better limb reconstruction compared to LASR. Similarly, in swing experiment in Fig LIMR delivers better results in the knee region, surpassing BANMo.
Skeleton Comparision. Firstly, we attempt to define the conditions necessary for an effective skele- ton in the task of 3D dynamic articulated object reconstruction: (1) The distribution of bones should
8

---

Published as a conference paper at ICLR 2024
be logical and tailored to the object's movement complexity discussed in the above paragraph (2) The skeleton must be detailed enough to accurately represent the object's 's structure while avoiding excessive complexity that might arise from local irregularities. As shown in Fig 4 we compared the skeleton results from RigNet Xu et al. (020)) (c) with our skeleton (a) for DAVIS-camel dance. Despite RigNet employing 3D supervision, including 3D meshes, ground truth skinning weights, and skeletons, it fails to achieve better skeleton results than LIMR, which operates without any 3D supervision. For instance, RigNet tends to assign an excessive number of unnecessary bones to areas with minimal motion, such as the torso of a camel. Additionally, in the results for dance, it lacks skeletal structure in crucial areas like the right lower leg and the left foot.
Skeleton Initialization. We learn the skeleton by first leveraging mesh contraction to get an initial skeleton with a larger number of bones and then updating the skeleton according to the SIOS² algorithm. As shown in 7 the initial skeletons always contain outlier points due to the unsmooth mesh surface, while our algorithm can effectively remove such noisy points during training.
Different Thresholds for skeleton Refinement. Throughout the skeleton updating process, we observed notable stability of our results vs minor variations in the threshold la around 0.5x current bone length, but a pronounced sensitivity to changes in lo- As depicted in 4 (a), LIMR tends to retain more bones and a more complex skeleton with higher to. This also leads to more semi-rigid parts in decomposition. Empirically, we observed marginal differences in reconstruction results when to lies between 0.99 and 0.85. However, excessively large or diminutive threshold values lead to too many or fewer bones resulting in poor reconstructions.
Impact of Video Content on skeleton. Our method intrinsically derives a skeleton by relying on the motion cues presented in the input videos. Consequently, varying motion contents across videos lead to distinct skeletons. Illustrativll,, in Fig (b), when provided with all 10 videos from human- cap, which includes actions within arms and legs, we learn a skeleton composed of 10 bones, allocating 2 bones for each leg. Conversely, using selected subset of videos where leg motions are conspic- uously absent, our system recognizes the leg as semi-rigid component, assigning a single bone for each. Also, the quality of the skeleton is influenced by the content within the video. For instance, in Fig the skeleton for human-cap cap appears to be of higher quality compared to swing. This is attributed to the diverse motions present in human cap, which includes actions such raising hands, leg movements, and squatting. Conversely, swing showcases a more simplistic motion.
Efficacy of Dynamic Rigidity and Part Refinement. We highlight the performance enhancements brought by introducing Dynamic Rigidity (DR), in contrast to the ARAP loss in contemporary works. As delineated in Tab[[ the integration of DR in approach yields performance boost of 1.7% on the DAVIS dataset and 1.7% improvement on the PlanetZoo dataset. Qualitative compar- isons are shown in Fig 9 As illustrated in F leveraging the learned skinning weights enables localized optimization for articulated objects, markedly enhancing the reconstruction fidelity of spe- cific regions. Notably, the leg reconstruction outcomes in both the camel and zebra experiments witnessed substantial improvements post the part refinement procedure.
CONCLUSION & LIMITATIONS
To conclude, we have introduced a method to simultaneously learn explicit representations (3D shape, color, camera parameters) and implicit representations (skeleton) of moving object from one or more videos without 3D supervision. We have proposed an algorithm SIOS² that iteratively estimates both representations. Experimental evaluations demonstrate the use of the 3D structure captured in the implicit representation enables LIMR to outperform state-of-the-arr methods.
Following are some limitations of LIMR. (1) Since it starts with a simple sphere instead of pre- defined shape template, it requires input videos to include diverse views for the best results. (2) Most of the available solutions, including LIMR, suffer from the problem of continuously moving cameras, which requires continuous estimation of the changing camera. Any errors therein propa- gate to errors in shape estimates since the model learns the shape and camera views simultaneously. For example, a poor shape estimate may lead to wrong symmetry plane. We plan to improve the model's 's resilience to camera-relate errors. (3) Our model, like many others, requires long training (10-20 hours per run A100 GPU with 40GB), and we plan working reduce it.

---

Published as a conference paper at ICLR 2024
REFERENCES
Oscar Kin-Chung Au, Chiew-Lan Tai, Hung-Kuo Chu, Daniel Cohen-Or, and Tong- Yee Lee. Skele- ton extraction by mesh contraction. ACM transactions on graphics (TOG), 27(3):1-10, 2008.
Mare Badger. Yufu Wang, Adarsh Modh, Ammon Perkes, Nikos Kolotouros, Bernd Pfrommer, Mare F Schmidt, and Kostas Danilidiis. 3d bird reconstruction: a dataset, model, and shape recovery from a single view. In European Conference on Computer Vision, pp. 1-17. Springer, 2020.
Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla. Creatures great and smal: Recovering the shape and motion animals from video. Computer Vision-ACCV 2018: 14th Asian Conference Computer Vision, Perth, Australia, December 2-6, 2018, Revised Se- lected Papers, Part V 14, pp. 3-19. Springer, 2019.
Benjamin Biggs, Oliver Boyne, James Charles, Andrew Fitzgibbon, and Roberto Cipolla. Who left the dogs out? 3d animal reconstruction with expectation maximization in the loop. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 195-211. Springer, 2020.
Junjie Cao, Andrea Tagliasacchi, Matt Olson, Hao Zhang, and Zhinxun Su. Point cloud skeletons via laplacian based contraction. In 2010 Shape Modeling International Conference, pp. 187-197.
IEEE, 2010.
Michael Garland and Paul S Heckbert. Surface simplification using quadric error metrics. In Pro- ceedings of the 24th annual conference on Computer graphics and interactive techniques, pp- 209-216, 1997.
Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoint without keypoints, In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pp. 88-104. Springer, 2020.
Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Point2mesh: A self-prior for de- formable meshes. arXiv preprint 1rXiv:2005.11084, 2020.
Yoonwoo Jeong, Scokjun Ahn, Christopher Choy, Anima Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 5846-5854, 2021.
Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 371-386, 2018
Ladislav Kavan, Steven Collins, Jifí Zára, and Carol O' O'Sullivan. Skinning with dual quaternions. In Proceedings of the 2007 symposium Interactive 3D graphics and games, pp. 39 46, 2007.
Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. End-to-end learning of geometry and context for deep stereo regression. In Proceedings of the IEEE international conference on computer vision, pp. 66 66-75, 2017.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafisnn, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. anythingg arXiv:2304.02644 026433 2023.
Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5253-5263, 2020.
Tianshu Kuai, Akash Karthikeyan, Yash Kant, Ashkan Mirzaci, and Igor Gilitschenski. Camm: Building category-agnosti and animatable 3d models from monocular videos, 2023.
Nilesh Kulkarni, Abhinav Gupta, David F Fouhey, and Shubham Tulsiani. Articulation-awall canonical surface mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 452-461, 2020.
10

---

Published as a conference paper at ICLR 2024
John P Lewis, Matt Cordner, and Nickson Fong. Pose space deformation: unified approach to shape interpolation and skeleton-drive deformation. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. pp. 811 1-1888. 2023.
Xueting Li, Sifei Liu, Shalini De Mello, Kihwan Kim, Xiaolong Wang, Ming-Hssun Yang, and Jan Kautz. Online adaptation for consistent mesh reconstruction in the wild. Advances in Neural Information Processing Systems, 33: 13000099-55 2020a.
Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Hsuan Yang, and Jan Kautz. Self-supervisee single-view 3d reconstruction via semantic consistency. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 677-693. Springer, 2020b.
Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8456-8465, 2023.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6498-6508, 2021.
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey, Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5741-5751, 2021.
Lingjie Liu, Mare Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt, Neural actor: Neural free-view synthesis of human actors with pose control. ACM transactions on graphics (TOG), 10(6):1-16, 2021.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the IEEE/CVF International Conference Com- puter Vision, pp-
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre- training for open-set object detection, 2023.
Diogo C Luvizon, Hedi Tabia, and David Picard. Human pose regression by combining indirect part detection and contextual information. Computers & Graphics, 85:15-22, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Atsuhiro Noguchi, Umar Igbal, Jonathan Tremblay, Tatsuya Harada, and Orazio Gallo. Watch it move: Unsupervised discovery of 3d joints for re-posing of articulated objects, 2022.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5865-5874 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin Brualla, and Steven M Seitz. Hypemerf: A higher-dimensioonll representation for topologically varying neural radiance fields. arXiv preprint 1rXiv:2106.13228 2021b.
Federico Perazzi, Jordi Pont-Tuset, Brian Mc McWilliams, Luc Van Gool, Markus Gross, and Alexander Dorkine-Hornung A benchmark dataset and evaluation methodology for video object segmen- tation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp- 724-732, 2016
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10327- 2021.

---

Published as a conference paper at ICLR 2024
Olga Sorkine, Daniel Cohen-Or Or, Yaron Lipman, Mare Alexa, Christian Rössl, and H-P Seidel. Laplacian surface editing. In Proceedings of the 2004 Eunographics/ACM SIGGRAPH sympo- sium Geometry processing, pp. 175-184, 2004.
Robert W Summer, Johannes Schmid, and Mark Pauly. Embedded deformation for shape manipula- tion. In ACM siggraph 2007 papers, pp. 80-es. 2007.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. pp. 402-419. Springer, 2020.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Con-
ference Computer Vision, pp.
2021.
Shubham Tulsiani, Nilesh Kulkarni, and Abhinav Gupta. Implicit mesh reconstruction from unan- notated image collections. arXiv preprint arXiv:2007.08504, 2020.
Daniel Vlasic, Ilya Baran, Wojeiech Matusik, and Jovan Popović. Articulated mesh animation from multi-view silhouettes. Acm Siggraph 2008 papers, pp. 1-9. 2008.
Kentaro Wada. Labelme: Image Polygonal Annotation with Python. URL httppp:::////tt
Com/ wwwetaaoo/labelllee
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 10689, 2021a.
Xiaohuan Corina Wang and Cary Phillips. Multi-weight enveloping: least-squares approximation techniques for skin animation. Proceedings of the 2002 ACM SIGGRAPHH/Eungraphics sym- posium Computer animation, pp. 129-138. 2002.
Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2 2021
Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CV conference computer vision and pattern Recognition, pp- 16210-16220, 2022.
Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magicpony: Learning articulated 3d animals in the wild, 2023a.
Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magicpony: Learning articulated 3d animals in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8792-8802, 2023b.
Yuefan Wu, Zeyuan Chen, Shaowei Liu, Zhongzheng Ren, and Shenlong Wang. Casa: Category- agnostic skeletal animal reconstruction. Advances in Neural Information Processing Systems, 35: 18559-285744 2022.
Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. Monocular total capture: Posing face, body, and hands in the wild. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10965-10974, 2019.
Zhan Xu, Yang Zhou, Evangelos Kalogerakis, Chris Landreth, and Karan Singh. Rignet: Neural rigging for articulated characters. arXiv preprint arXiv:2005.000 .0550, 2020.
Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from a monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15980-15989 2021a.
12

---

Published as a conference paper at ICLR 2024
Gengshan Yang. Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan. Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. Advances in Neural Information Processing Systems, 34:19326- 19338, 2021b.
Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2863-2873, 2022.
Gengshan Yang. Chaoyang Wang, N Dinesh Reddy, and Deva Ramanan. Reconstructing animatable categories from videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16995-17005, 2023.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. Ad- vances in Neural Information Processing Systems, 33:2492-25022 2020.
Hao Zhang, Fang Li, Lu Qi, Ming- Yang, and Narendra Ahuja. Csl: Class-agnostic structure- constrained learning for segmentation including the unseen. Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 38, pp. 1078-7086, 2024.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 386-595, 2018.
Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black. 3d menagerie: Modeling the 3d shape and pose of animals. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6365-6373, 2017.
Silvia Zuffi, Angjoo Kanazawa, and Michael Black. Lions and tigers and bears: Capturing non- rigid, 3d, articulated shape from images. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 3955-3963, 2018.
Silvia Zuffi, Angjoo Kanazawa, Tanya Berger- Wolf, and Michael Black. Three-d safari: Learning to estimate zebra pose, shape, and texture from images" in the wild". In Proceedings of the JEEF/CVF International Conference on Computer Vision pp. 5359-5368, 2019.
13

---

Published as a conference paper at ICLR 2024
Shape Template
Skeleton Template
Motion Manipulation
Multiple
Camera Pose
Learn
Method
Videos
Skeleton
LASR Yang et
2021a
X
virtual
x
X
VISER
Yang et
virtual
BANMO
2020
virtual
CASA Wa al. 2022
V
physical
x
X
WIM Noguchi of al. 2022
X
X
physical
v
V
CAMM Kuai et 2023
K
physical
X
RAC Yang cl. (2023
physical
X
MagicPony
physical
Ours (LIMR)
physical
Table 2: Difference between LIMR and existing methods
A APPENDIX
In this section, we provide: (1) More information about the mesh contraction operation A.1
Details of the losses and regularization used in the NeRF-basee scheme (A.2.2) and NMR-based scheme (A.2.I). (3) Additional qualitative results (for rendering and mesh) for different videos in the PlanetZoo dataset in Fig 8 9 (4) Quantitative results for AMA's swing and samba in
Tab (5) Details of test datasets in A.4 (6) Mesh results during skeleton updating Fig
Difference between LIMR and existing methods in Tab and See A.6 (9) More discussion of the difference between two explicit representation
(7) Details of the SIOS² algorithm in Alg
schemes in A.5 and failure cases due wrong camera predictions in A.5 (10) More discussion
about our bone motion estimation strategy compared with WIM Noguchi et al. 2002).
(11) Nota-
tions in Tab
Algorithm Synergistic Iterative Optimization of Shape and Skeleton (SIOS?)
1: Initialize the parameters ((0) for the reconstruction model and extract the initial skeleton ST (0)
from the current rest mesh M utilizing mesh contraction (Sec 3.1
2: for 0, 1,2,.... until convergence do
E-Step
3: Transform ((e) to time-space M. by integrating the transformations of bones (Sec 3.1
4: Render and compute the reconstruction losses (Sec 3.2)
5:
Update the parameters in the reconstruction model: g 1) g(() (Sec 3.3
D M-Step
6:
Randomly sample H images from all frames.
7:
Compute the bone motion direction in the selected frames using optical flow warping
(.1)
8: Determine joint coordinates and bone lengths in the selected frames (Sec 3.1
9: Refine the skeleton considering physical constraints: Sy (e+1) <- ST (e) (Sec 3.3
end for
A.1 MESH CONTRACTION
Laplacian Contraction, driven by weighted forces, creates thin skeleton representing the object's logical components. The contracted vertex positions, X', are determined by Laplace equation:
LX' 0. Here, L is the curvature- flow Laplace operator defined as:
Wij cot asj cot Big Σ
if if. E E
Lij
(L,))
WW
if = otherwise.
(5)
and aij and By the opposite angles corresponding to the edge (i, Then, minimize the following quadratic energy:
WcLX
+
Σ W W X
1
(6)
14

---

Published as a conference paper at ICLR 2024
with diagonal weighting matrices We and Wa, A. balancing contraction and attraction, the i-th diag- onal element of W is denoted W A. and the coordinate of vertex denoted X,. Subsequently, we update C = sz W and W A.i W V VA..VA,// A where A and A are the current and the original one-ring areas, respectively. With the updated vertex positions, a new Laplace operator, L is computed. We use the following default initial setting: W = 1.0 and W 11-1100-00 VvA, where is the average face area of the model.
A.2 LOSSES AND REGULARIZATION
A.2.1 NERF-BASED SCHEME LOSSES
In this section, we explain the losses in the NeRF-based scheme in detail. Additional details available in BANMo Yang et al. (2022).
C: MLP (X,, U/, wi))
σ (MD (X.))
(8)
S (LP.,(X.)))
(9)
Here x means the 2D projection of X
Reconstruction Loss:
Σ lo(x')) - 0(x0)12
RRGB
(10)
Lxilhouetle =
Σ s(x')) - s(x)
(11)
Loptical-Flow
F.St
1...t
(12)
x
We compute CRGB, Lailhouctee Loptccl-Flow all by minimizing the L2 norm of the difference between the rendered outputs and the ground truth. c(x'), oxx'), FS respectively represent the ground truth color, silhouette and optical flow at time and e(x') 8(x") f.S.* mean the rendered ones at time correspondingly.
Feature Loss & Regularization Term
Cieature embedddig
Σ 8x(xx)) - X'
(13)
Σ (x' -x12
C2D-matching
(14)
L3D-consistency
Σ | (W (x)) -
(15)
W and ype represent the Blend Skinning and the Backward Blend Skinning involving time t in 3.1 т is defined as opacity to give more regularization on the points near the surface. (x") and X* (x') are respectively the canonical embeddings in prediction and backward warping by soft argmax descriptor matching Kendall et al. (2017); Luvizon et al. (2019) at time t. II' the projection matrix time /. from 3D to 2D. For more details refer to BANMo Yang al. (2022)
15

---

Published as a conference paper at ICLR 2024
AMA-samba
AMA-swing
Ave.
method
CD
F@22%
mloU
CD
F@2%%
mloU
CD
F@ 2%
mloU
BANMo
15.3
53.1
61.2
13.8
54.8
62.4
14.6
53.9
61.8
Ours
13.1
55.4
61.8
12.7
56.2
63.2
12.9
55.8
62.5
Quantitative results on AMA's swing ng and samba. 3D Chamfer Distance (cm, 4), F-score (% 1), and mIoU (% shown averaged over all frames. Note use half batch size compared with BANMo Yang et al. (002)). which leads to lower accuracy.
Table 3:
Input
LASR
Ones
Ingut
LASR
Ours
Figure 8: Rendering Results on PlanetZoo Dataset. Here we compare our approach with LASR on PlanetZoo's dog and zebra.
A.2.2 NEURAL MESH RASTERIZATION-BASE SCHEME LOSSES
We define St It 22D the silhouette, input image, and optical flow of the input image, and their corresponding rendered counterparts (St. Ît. F2D.1)
Reconstruction Loss: For an NMR-based scheme, we write the total reconstruction loss as a com- bination of silhouette loss, texture loss, optical flow loss, and perceptual loss (pdist).
Creconstmation =13 St -st| 1t
32 [ - Ie 330 ( - (F2..)) + Supdist( I')
(16)
Here, I's are the same with LASR, and is the normalized confidence map for flow.
Shape Loss:
To ensure the smoothness of the mesh, Laplacian smoothing applied Sorkine al. 2004 The
smoothing operation is described per vertex as shown in Equation 17
Shaee
X
x JEN,
(17)
N
Where, X is coordinates of vertex i in canonical space.
A.3 IMPLEMENTATION DETAILS
A.3.1 NERF-BASED SCHEME
Even when frames available from enough different viewpoints, directly extracting the mesh sur- face from the NeRF field will result in an inconsistent and non-smooth surface. So closely following BANMo, utilize the Signed Distance Function (SDF) described in Eq[ To model dynamic
16

---

Published as a conference paper at ICLR 2024
Wilsout
Figure 9: Reconstruction Outcomes w/ and w/o Dynamic Rigidity. We present the reconstruc- tion outcomes with and without Dynamic Rigidity on PlanetZoo's elephant, bear, and dog sequences.
scenes, in addition to color ét, view direction v², and canonical embedding v E RR6,, time variable should also be included to account for deformations along time The canonical embedding is de- signed to adapt to the environmental illumination (w)). The MLPsDF efficiently models the point outside the surface, the SDF value is negative, and vice versa. 7(x) is zero-mean and unimodal distribution accumulating the output from MLPsDF transfer the SDF value density a. The zero level-set of the SDF value makes up the extracted surface.
X, the signed distance the surface. If the points
To ensure fair comparison, our experiments keep most of the optimization and experiment details the same as BANMo Yang et al. (2022). We extract the zero-level set SDF values in the neural radiance field by marching cubes predefined 256 grid. Different from BANMo, we initialize the potential rest bones by mesh contraction (Sec only one time after the warmup step and allow synergistic iterative optimization (S.e to refine the locations and number of bones. AdamW is implemented as the optimizer with 256 image pairs in each batch and 6144 sampled pixels. We train the model on one A A100 40GB GPU and empirically find the optimizations stabilize around the 5th training step with 1h time cost and 15 epochs for each step. The learning rates are set up by a 1-cycle learning rate scheduler starting from the lowest ITiit
x 10 to the highest value Irmax 5 x 10 and then falls to the final learning rate |rinal 1 x 10 The rules of Near-far plane calculations and multi-stan stage optimization follow the same settings as BANMo.
A.3.2 NEURAL MESH RENDERER -ASEDD SCHEME
The Neural Mesh Rasterization (NMR)- based approaches directly learn the mesh and potential cam- era poses rather than extracting them from the neural radiance field of the NeRF-based one. The rest mesh is either provided or initialized by projecting subdivided icosahedron onto sphere and fur- ther refined (deformed) through a coarse-to- fine learning process. The refinement and deformation carried out using blend skinning 3.1 In every single forward pass, employ soft-rasterization to render 2D silhouettes and color images for self-supervised loss computations. This rendering identifies the probabilistic contribution of all the triangles in the mesh to the rendered pixels.
17

---

Published as a conference paper at ICLR 2024
Figure 10:
Annotations for the PlanetZoo Dataset. Following the annotation guidelines es- tablished in BADIA, we have annotated our newly collected PlanetZoo dataset. Consistent with BADJA, annotations are provided every five frames.
We learn the time-varying parameters, including the transformation T and camera parameter Pc. similar to LASR. To generate segmentation masks for the input videos, we used the Segment Anything Model Kirillov et al. (2023). Optical flow was estimated using flow estimators [Tedd & Deng (2020). Furthermore, we employ the coarse-to-fine refinement approach used by Point2Mesh Hanocka et al. (2020) and LASR Yang et al. (2021a), but there is a crucial difference: in the pro- cess of refinement, previous approaches fix the number of bones B for each stage and empirically increase the value when moving on to the next stage of the coarse-to-fine refinement. In our case, we start with a large value of B defined from the bones in the initial skeleton. We use the surface flow direction FS reduce the value of B. We iteratively perform this process and end up with the most optimized positions of the bones, which further leads to part decomposition W. For our experiments, use A100 40GB GPU. We set the batch size 4, and epochs are kept the same as LASR.
A.3.3 MASK AND OPTICAL FLOW GENERATION.
We generate object-level segmentation masks for input videos using Grounding DINO Liu et al. 2002),, CSL Zhang al. (2024) and Segment Anything Model (SAM) Kirillov et al. (2023). ). By inputting text about the target object in the video, we leverage Grounding DINO to track that object across frames and then input the detected bounding boxes SAM/CSL obtain the segmenta- tion masks. Optical flow was estimated using existing flow estimators Teed & Deng (2020).
A.4 DATASETS AND METRICS
We demonstrate LIMR's performance on various public datasets including typical articulated objects like humans, reptiles, quadrupeds, etc. To evaluate LIMR and compare our results with previous works, we also adopt two quantitative metrics besides the conventional qualitative visual results like the rendered 3D reconstruction shapes, skeletons, and part assignments. The first one is 2D keypoint transfer accuracy Yang et al. (2021a) meaning the percentage of correct keypoint transfer (PCK-T) Kanazawa et al. 221 8); Kulkarni et al. (2020). Given the ground truth 2D keypoint annotations, we label the transferred points whose distance to the corresponding ground truth points is within the threshold as 'Correct'. This threshold is defined as dm = /ST. where |S| represents the area of ground truth mask Biggs al. (2019). The other is the 3D Chamfer distance Yang et al. (2022) which averages the distance between the rendered and ground truth 3D mesh verüices. The matches between each pair of vertices set up by finding the nearest neighbor.
18

---

Published as a conference paper at ICLR 2024
Figure 11: Reconstruction Outcomes with Different Number of Bones. We present the recon- struction outcomes with different numbers of bones on PlanetZoo's elephant sequences.
DAVIS & BADJA. Biggs et al. (019) derives from DAVIS video segmentation dataset et al. (2016) containing nine real articulated animal videos with ground truth 2D keypoints and masks. includes the typical quadrupeds like dogs, horses, camels, and bears. In addition to the visualization performance, we also adopt the quantitative 2D keypoint transfer accuracy given the ground truth.
PlanetZoo is made up of the virtual animal videos collected online. To show our generalization on articulated objects, we collected more quadruped videos and reptile videos with much larger move- ments compared with the existing public datasets. For example, the animals move much quicker and more complex, and the limbs of dinosaurs with largely different lengths and morphologies are not easy for the existing methods Wu et al. (2022) to the predesigned per-category ground truth skeletons. We annotated the reasonable ground truth 2D key points using Labelme tools Wada fol- lowing the real physical morphology across all frames in each collected animal video so that the 2D key points transfer accuracy can be evaluated. The annotation configurations and samples are shown in Figure I0] 2D annotated key points are located on the paw(hand), nose, car, jaw, knee, tail, neck, elbow, buttock, thigh, and calve, which are common for most animals following morphology. Especially for tails, most tails are long and move flexibly, so the 2D key points are annotated along the tails on the middle point and two ends of the tails.
AMA human & Casual videos dataset. Vlasic al. (2008) records multi-view videos by synchronized cameras. Ignoring the ground truth synchronization and camera parameters, only RGB videos are used in optimization and the experiments only take the decent ground truth 3D meshes in use for calculating the 3D Chamfer distance. The casual videos of
and Human- caaman-capaaa are collected by BANMO Yang et al. (2022) following the same way with AMA datasets but without ground truth meshes.
A.5
MORE DIAGNOSTICS
Bone Motion Estimation. In WIM Noguchi et al. (2022), objects are initially represented as set of parts, each approximated with an ellipsoid. The method involves learning the pose of each part and then deciding whether to merge two parts based on the relative displacement observed between them throughout the video. In this context, each part is considered a rigid body, having the same SE(3), which allows for the calculation of relative positions by comparing the SE(3) of adjacent parts. However, unlike WIM, where each part is treated as a rigid entity, our approach treats each part semi-rigid motions, but with allowances for minor variations.
This means that the surface vertices within the same part exhibit similar
We implement this concept using blend skinning techniques. The process involves learning the SE(3) of B bones and the skinning weights that describe the relationship between these B bones and surface vertices. By calculating the linear combination of the SE(3) of bones based on skinning weights, we determine the SE(3) for each vertex. This approach allows for more flexible descrip- tion of each vertex's motion.
Intuitively, we aim to correlate each part with single bone, but this one-to-one correspondence often does not hold true. In practice, we find that skinning weights are typically non-sparse, with the motion of surface vertices being determined by multiple bones. Consequently, the motion of each semi riiid part is usually influenced by several nearby bones, especially during the initial stages af- ter skeleton initialization when the number of bones is large. In such cases, representing the motion of one part with a single bone becomes unreasonable, a fact supported by our initial attempts. To address this, consider using 2D optical flow determine the 2D motion of surface vertices
19

---

Published as a conference paper at ICLR 2024
and skinning weights to calculate the 2D motion eacc part. This process helps in deciding whether adjacent parts or bones should be merged. Here, 'bone' refers to the one with the highest weight in relation to the part, but it might not be the only bone affecting the part's motion. Our experiments demonstrate that this method yields satisfactory results.
NeRF-based methods VS NMR-based methods NeRF-based methods such as BANMo have showcased impressive reconstructions. These approaches involve the learning of a neural radi- ance field, from which a mesh is extracted. However, learning such a field requires a substantial amount of data, typically including multiple videos and viewpoints, which can be challenging to acquire. These approaches experience a significant drop in performance when provided with lim- ited data and views as shown in F On the other hand, works based on neural mesh renderer directly train models to generate a mesh, yielding satisfactory results in limited data settings, such as single monocular video with fewer than a hundred input frames. We demonstrate the applicabil- ity of our approach both of these scenarios and showcase significant qualitative and quantitative performance improvements in cases with limited data.
Failure Cases due to Wrong Camera Predictions. BANMo exhibits high sensitivity to extensive camera motion within videos. This is attributed to its reliance PoseNet to estimate the camera pose from the initial frame. Significant pose variations in subsequent frames can severely degrade its performance. For instance, shown in Fig in the zebra experiment, it learns two heads incorrectly. As illustrated in Fig 13, the Symmetry Loss adversely affected by videos exhibiting long-range camera motions. Incorrect camera parameter predictions hinder the algorithm's ability to deduce a plausible symmetry plane. [3 displays various views of the reconstructed zebra, highlighting the inaccuracies in the symmetry plane. Consequently, we opted to exclude the sym- metry constraints.
A.6 DIFFERENCE BETWEEN LIMR AND EXISTING METHODS
As shown in Tab. 2 we list the comparison among LIMR and existing methods regarding different settings. LIMR tackles more challenging but realistic problems which are: 1) In the wild, there are rarely ground truth (GT) 3D shapes, skeleton templates, and camera poses provided. 2) We cannot ensure multiple videos of moving objects containing different actions and views can be provided. Works Kuai et al. (2023): ; Yang et al. 2023): Wu al. (2023)) like CASA Wu et al. (2022), and SMAL Zuffi et al. (2017) requiring instance-speciffed 3D shapes/skeleton/both templates of each object for motion modeling will be undoubtedly limited by their generalization to other Out-Of- Distribution objects without GT 3D information as input. Some approaches such as WIM Noguchi al. (2022). CAMM Kuai al. (2023). RAC Yang al. (2023), and BANMo Yang al. (2022) require providing accurate camera poses and multiple videos (more than 1000 frames) with diverse views in order to provide decent results. Methods like LASR Yang et al. (2021a) and VISER Yang et al. 2021bb do not require a pre-built template as input. However, due to the absence of a skeleton that can provide structural information about the object, they often fail to achieve optimal results. In contrast, LIMR used the learned near mear-physical skeleton to facilitate modeling the motions of moving articulated objects instead of using virtual bones in LASR Yang et al. (2021a), BANMo Yang et al. 2022) and ViSER Yang al. (2021b), while significantly minimizing the requirements for input.
20

---

Published as a conference paper at ICLR 2024
Swing
Samba
Samba
Swing
Update Epach
Skeletan Update Epoch
Figure 12: Reconstruction Outcomes during Skeleton Updates. present the reconstruction outcomes at various epochs post-skeleton update for AMA's swing and samba sequences.
Figure 13: Failure Cases due to Symmetry Loss.
21

---

Published as a conference paper at ICLR 2024
Table 4: Table Notations
Symbol
Description
Dimension
B
Number of Bones
E
Number Edges in Surface Mesh
J
Number of Joints
Representations Notations
Re
Explicit Representations
R;
implicit Representations
Representation Parameters
M
Canonical Surface Mesh and Color
P
Camera Parameters
W
Skinning Weights
W WE E NXB R E RE TE SE(3)
R
Rigidity Coefficient
T
Time Varying Transformation
Skeleton Notations
ST
Skeleton
B
Bones
B Rx13 JER
JJ
Joints
Mest Contraction
X
Vertices Coordinates
E
Edges between Vertices
Properties of 3D Points
Color of 3D point
C R E R
Density of a 3D point
vb
Canonical embedding of a 3D point
1b E R16
Bone Components
C/B(t,
Gaussian Centers Coordinates
C E R Bx3
Q/B[: 12]
Precision Matrix
Q E RR Bx9
L/B |, 12]
Bone Length
R
Joint Components
1;:2 J 2
Index of Two connected Bones
RJx2 R
Joint Coordinates
Optical Flow Notations (at time f)
FED.i
2D Optical Flow
FED. R
FB.
2D Bone Motion Direction
FB. E R Bx2
Fs.
Surface Flow Direction
FSA E RXx
V
Visibility Matrix
Vi E Nx11
Blend Skinning Functions
W (X") (X
Forward Blend Skinning from X to X
Backward Blend Skinning from X' to X
2D Notations
S/S I/i 22D
Input silhouette/ Observed silhouette
S/S E B I/ E R 22DD e "xw
Input Image / Observed Image 2D Input OF 2D observed OF
22


Reviews:
[{"review_id": "5PdygJWM03", "reviewer": "Reviewer_Z1Zw", "summary": "This paper aims to reconstruct the 3D shape of the moving articulated object from one or multiple monocular videos. The paper proposes LIMR (Learning Implicit Representation) that models both explicit information (3D shapes, colors, camera parameters) and implicit skeletal information. To iteratively estimate both implicit and explicit representations, the paper proposes Synergistic Iterative Optimization of Shape and Skeleton (SIOS2) algorithm that uses physical constraints as regularization terms. Experiments on standard datasets show that LIMR outperforms state-of-the-art category-agnostic methods.", "questions": "Some suggestions on the clarity of presentation:\n1. Currently, the figures are not placed in an ascending order (Figure 7 before Figure 5) and the references to the figures in the text jump back and forth.\n\n2. Some notifications are not well explained, eg, N_i in Eq. (3).\n\n3. Typo: \"(3)\" in the first paragraph of appendix A: \"quantitative\"-->\"qualitative\".", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a novel method for the important task of reconstructing moving articulated object from monocular videos. The proposed joint explicit and implicit representations seem effective in modeling both canonical structure and pose-dependent deformation.\n\n2. Experiments have been conducted for a fair comparison with state-of-the-arts (i.e., LASR and BANMo) that do not take ground truth skeletons. The experiments include both qualitative (geometry and appearance) and quantitative (2D keypoint transfer and 3D shape reconstruction).\n\n3. Extensive ablation studies have been conducted to show the importance of each component in Sec. 4.3. \n\n4. Limitations and implementations have been discussed in detail. For example, one limitation is that the proposed method requires 10-20 hours to learn, which are comparable with baselines (LASR and BANMo on the order of a few hours).", "weaknesses": "1. More visualizations such as video comparisons like those shown in LASR and BANMo would be more intuitive and straightforward to show the object in the move/motion.\n\n2. The proposed system includes multiple hyper-parameters and multiple separate but interdependent steps. Given the complications of the current system, it is unclear how robust this method is, eg, with regard to initialization, hyper-parameter settings, input video contents, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dd8PSIuCNY", "reviewer": "Reviewer_dR3E", "summary": "This paper proposes an algorithm to reconstruct a semi-nonrigid articulated object from RGB video. The geometry is modeled by an explicit mesh and the motion is modeled by a learnable skeleton and skinning. The geometry and motion models are jointly optimized on the video by flow supervision and several prior regularization losses. Experiments show that the proposed method can reconstruct animals and humans from the DAVIS dataset.", "questions": "See the weakness. The main question is the motivation for studying such a task in the way this paper presented.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method somehow worked on DAVIS to reconstruct Quadrupeds and the Human body.\n- The idea of learning skinning and differentiable skeletons is interesting, this may inspire other deformation representations and general dynamic scene modeling. But not in this task (see weakness)", "weaknesses": "- The main concern lies in the necessity and value of the task in the current literature. There are two aspects to argue this:  1.) The reviewer guesses that given the current SoTA and technology in the community, the best way to model semi-nonrigid objects presented in this paper is to use template-based models. This paper presents animals and humans, which already have good template models. Only when the object motion structure differs a lot, and lacks a good template model, do we need some “unsupervised” algorithm to find the structure from videos. However, the paper never presents an example of such a case. 2.) The other way to recover the articulated object from the RGB (no depth) video is to first totally forget articulation and treat the scene as general dynamic 4D functions. In this way, given the current advanced dynamic rendering and reconstruction from monocular video, one can easily get the geometry as well as long-term correspondence first and then segment and easily extract the articulated object. Given these facts, I currently don’t believe this paper’s direction makes a real contribution. But as I write in the strengths, the idea of differentially learning the motion structure is interesting but needs to be carefully put into the right context and motivated nicely.\n- The method is somehow complicated and lacks principles when presenting, making it not very easy to follow. Given a complex algorithm like this, I can hardly tell which component is really important, although some ablations are provided, given the complexity of the model, these may not be enough.\n- The quality of the reconstruction shown in the figure is not appealing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4r5R4a7VIy", "reviewer": "Reviewer_gF7P", "summary": "This paper focuses on learning non-rigid articulated objects from monocular videos. The proposed method employs a template surface mesh (explicit representation) and a skeleton (implicit representation) simultaneously without category-specific pretraining, followed by the proposed alternating optimization approach of surface mesh and skeleton. Additionally, a refinement method for the skeleton during optimization is introduced, allowing for adjusting the number of joints during the optimization. The method also introduces a part refinement technique, enhancing limb reconstruction. The proposed method is primarily compared with BANMo, and LASR, and evaluated across multiple benchmark datasets, demonstrating qualitative/quantitative improvements.", "questions": "- How would the result change without considering optical flow visibility and Laplacian contraction?\n- How should we interpret the improvement by rigid dynamics from Fig.9?\n- Is there any reason more recent methods like CASA/MagicPony are not compared with the proposed work?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method does not require category-specific pre-training, unlike MagicPony and BANMo.\n- The proposed method demonstrates significant qualitative improvements over prior works.\n- The proposed method incorporates a novel mechanism for adaptively learning the optimal number of skeleton joints.\n- Extensive ablation studies are conducted.", "weaknesses": "**Major**\n- Missing ablation of considering optical flow visibility (Eq. 4) and Laplacian contraction.\n- The effectiveness of the rigidity coefficient/dynamic rigid does not seem substantial from Fig. 9 and Table 7.\n- The most recent method, MagicPony, also employs implicit and explicit representation and strongly relates to the proposed work, yet a direct comparison in the experiment is missing.\n\n**Minor**\n- Although as a post-processing step, WIM [1] also infers the skeleton with a variable number of joints for articulated targets and the relation could be discussed.\n\n[1] Noguchi et al. Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects. CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LR2wDmNWxF", "reviewer": "Reviewer_2K9U", "summary": "The paper tackles the problem of articulate 3D shape reconstruction from a single video, similar to the setup of LASR. The main contribution is a method for automatic skeleton structure discovery. They also introduced a variant of local rigidity loss that accounts for the flexibility of surface points. \n\nThe method is evaluated on DAVIS and Planet Zoo datasets, out-performing LASR/ViSER. It also shows qualitative results on AMA and  BANMo's videos, with better performance than BANMo.", "questions": "1. Fig 1 caption: Is the rigidity coefficient a parameter? I thought R could be directly computed from D.\n2. It would be more convincing to include video results and comparisons.\n3. BANMo's result in Fig 3 appears particularly bad. Is there any explanation? The results on Swing also appear coarse. What is the resolution of the marching cubes? Also, I overlaying the skeleton with the mesh similar to Fig 4. would make the results more compelling.\n\nOther comments\n1. Fig 6 is nice as it shows the initial configuration of the skeleton. It would be nicer to show the progression of the skeleton over iterations.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Originality**\n\nThe paper introduces a few new techniques I found interesting. \n- The Dynamic Rigid (DR) loss aims to encourage local rigidity (similar to ARAP) in a more flexible way. Instead of applying ARAP uniformly on the surface, it is weighed by the \"Rigidity coefficient\", which is a function of skinning weights. In this way, edges with peaky skinning weight will receive more penalty as they should move more rigidly. The effect is validated in Tab 1. \n- They leverage 2D motion (flow) constraints for skeleton discovery. To achieve this, an inverse blend skinning method is used to aggregate a 2D flow map to bones. Bone with similar 2D flow will be merged.", "weaknesses": "**Presentation**\n- In general, I feel the presentation could be significantly improved and the paper could be made much stronger.\n- The usage of certains terms made it difficult to follow. For example: \n  - \"semi-rigid part assignment\" can be replaced by \"skinning weights\", which is a term that already exists in graphics and vision literature\n  - \"implicit motion representation\": I would think it is explicit representation, given they are actually a set of rigid transformations that are explicitly defined. I think \"Internal / latent\" representation better describes skeleton movements.\n  - \"inverse blend skinning\" (IBS) is an unconventional term and needs more explanation and highlighting. Indeed, one may find naming it inverse of blending skinning not accurate. Blend skinning maps a set of points X, rigid transformations G, and skinning weights W to another set of points X'=WGX. The inverse could be finding G and W from X'. My understanding is that the paper uses IBS to map vertex properties (e.g. 2D flow) to bones with an existing W. Is this correct?\n- The method section could be better structured with some \"pointer\" sentences. For example, bone motion direction is introduced in Eq (4) but it was not mentioned why we need to compute them. It only appears in Sec 3.3. With this, I would suggest either move IBS to Sec 3.3, or pointing the reader to Sec 3.3 when motivating Eq (4).\n- It is not immediately clear what purpose Sec 3.2 aims to serve. The rest of the paper describes the shape representation as meshes, but this section talks about the pros and cons of mesh vs neural fields. If the goal is to how that method can be applied to neural fields as well, I feel this can be moved to implementation details.\n\n**Related works**\n- Some related works are not discussed. \n  - [A]-[B] find skeleton from video\n  - [C]-[D] find skeleton from 3D point/mesh sequences\n  - [E] finds skeleton from image features\n- Since [A]-[D] also deal with motion data, some method-level discussion or even comparison is needed. I think [A] is particularly related as they also search for the skeleton structure and optimize shape in an alternating fashion.\n\n\nExperiments\n- To strengthen the experiments, the bone discovery results could be separately evaluated and compared against some existing work, such as Rignet or Morig[D].\n- For bone merging, one alternative is to use the similarity of SE(3) of bones. The effect of \"2D flow\" vs \"3D rigid transform\" could be ablated through an experiment.\n\n\n[A] Noguchi, Atsuhiro, et al. \"Watch it move: Unsupervised discovery of 3D joints for re-posing of articulated objects.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[B] Kuai, Tianshu, et al. \"CAMM: Building Category-Agnostic and Animatable 3D Models from Monocular Videos.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[C] Le, Binh Huy, and Zhigang Deng. \"Robust and accurate skeletal rigging from mesh sequences.\" ACM Transactions on Graphics (TOG) 33.4 (2014): 1-10.\n\n[D] Xu, Zhan, et al. \"Morig: Motion-aware rigging of character meshes from point clouds.\" SIGGRAPH Asia 2022 Conference Papers. 2022.\n\n[E] Yao, Chun-Han, et al. \"Lassie: Learning articulated shapes from sparse image ensemble via 3d part discovery.\" Advances in Neural Information Processing Systems 35 (2022): 15296-15308.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ELQk3i0jtL", "reviewer": "Reviewer_Wx9e", "summary": "Input:  One or more monocular video containing moving object that exhibits articulations; \nOutput: 3D shape of moving object in the input video\n\nThe paper presents an iterative synchronization framework that predicts the 3D shape of moving objects, specifically, living beings such as animals and humans, from input monocular videos. The 3D shape is decomposed into two parts – explicit (the surface and its color), and implicit (the skeleton, the object’s semi-rigid parts, their motions, and the articulated part motion parameters given by rigidity coefficients).\nIt presents a Synergistic Iterative Optimization scheme for Shapes and Skeletons, henceforth referred to as SIOS^2, that learns both explicit and implicit representations (as defined above), iteratively. This is analogous to the Expectation Maximization (EM) algorithm in Machine Learning. \n\nDuring the E phase, the implicit skeleton is improved based on the current explicit rep. In this step, the 2D motion direction of each semi-rigid part, i.e., the bone, is calculated and distances between connected joints at the end of the bone are measured. The skeleton is then updated using the consistency of direction of the optical flow within each semi-rigid part, and the constancy (not consistency) of distances between connected joints, i.e., bone length, across the video frames.\n\nWith this E step, the 3D shape is updated in the M step using the updated skeleton (obtained from the E step).\n\n\n\nDataset used:\n\nThe paper uses the following different kinds of datasets that contain videos of articulated living beings.\nDAVIS – Animals\nPlanetZoo – Animals\nAMA – Humans\nBANMO - Humans\n\n\n\nUnderlying Neural Network:\n\nFor the so-called “implicit representation learning”, i.e., skeleton-associated representation learning, I do not see any neural network being used. Pls confirm if this is this is the case. What I understand from the paper is that this stage builds upon existing techniques and uses non-neural network-based optimization to obtain Bones, Vertices and Joints.\n\nThe neural part is during the learning of explicit representation, which learns the surface of the articulated living being. This is where two different kinds of neural networks are used.\nThe first one is a NeRF model with MLPs on Signed Distance Fields (SDFs). The second model is a Neural Mesh Rasterizer (NMR) model that is used to enforce consistency in the 2D domain.\n\n\nLoss functions: \n\nSimilar to Volumetric-based rendering methods for 3D surface reconstruction (Yariv et al. 2021), the NeRF model uses the following loss terms:\nL_NeRF = L_recon + L_feature + L_3DConsistency, where\nL_recon = L_silhouetter + L_rgb + L_opticalflow\nL_feature is the 3D feature embedding loss that minimizes the distance between the canonical embeddings of the 3D shape from prediction and backward warping, and\nL_3Dconsistency  is used to ensure forward-deformed 3D points match their original location upon backward-deformation.\n\nL_NMR = L_recon&perceptual + (alpha x L_shape) + (lamda x L_DR), where\nL_recon&perceptual = is the same as L_NeRF but with additional perceptual loss,\nL_DR is the dynamic rigid loss and L_shape is the shape regularizer\n\n\n\nQuantitative Metric:\n\nTable 1- 2D Keypoint Transfer Accuracy, \nTable 3- Chamfer Distance on shape reconstruction task from multiple videos", "questions": "1) I am not sure if the skeleton representation should be termed an implicit representation. It is intrinsic/internal to the shape. In today’s context of neural fields, using “implicit rep” to refer to skeletonization is misleading.\n\n2) The minimum number of videos that can be input to the method is 1. What is the max? \n\n3) And is there a limit on the video length? What is the minimum number of frames in the input video that should contain the articulating living being for LIMR to work? Can it work if there are just two frames that contain the living being? Just one frame is not possible I believe?\n\n4) What is the resolution of the video input?\n\n5) What is the resolution of video frames processed by the LIMR framework? Same as video input or do you downsize?\n\n6) Can LIMR work with grayscale videos?\n\n7) What is the range of number of bones and bone joints (max and min) that this method can work on?\n\n8) Can videos containing multiple objects be used as input? If so, can all the objects be recovered in terms of their shapes? If not, which object will be selected in such a case? I guess the method will need an additional pre-processing step to detect object classes and/or add a saliency module. Can you comment on these? These are all important considerations that need to be discussed in the limitations section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper addresses a challenging task of estimating deformations on 3D shapes of humans and animals, in terms of both surface/skin and skeleton articulations, from one or more monocular videos. In doing so, no ground truth 3D supervision is used.\n\n2) The clarity of writing is reasonable, something that does not appear to be difficult to follow for folks working in this area. And most of the related works in the context of this paper are discussed to the degree that is possible within the page limit.\n\n3) The limitations are aptly discussed, which educate the readers about the gaps to be filled despite the attempts/contributions in the paper.\n\n4) The SIOS2 algorithm, which is an interleaving-iterative algorithm, is intuitive and showing that this works on this task (i.e., the task of estimating 3D shapes of living beings that undergo articulations, from video inputs) adds value to such co-analysis based techniques in Visual Computing research.", "weaknesses": "1) The presented pipeline is overly complicated, with the iterative SIOS2 algorithm coupled with optical flow estimation to constrain the prediction of shape skeleton based on consistency across different frames of the video. And so, the method may not be easily extendable to potential new projects in this direction. Plus, I have concerns about the reproducibility of the method. \n\n2) Another major concern I have is the lack of diversity of shape samples used to demonstrate the superiority of the reconstruction/rendering results from LIMR (the paper’s approach). Specifically, Fig 3 shows limited diversity of shape samples (a Camel and a Horse). And I have not seen much diversity in the Appendix either (Fox, Zebra and Elephant; I would consider a Zebra equivalent to a Horse). Is this because PlanetZoo and DAVIS datasets only have these classes of animals? \n\n3) The rendering results are poor in quality, see Fig 3 and 7. It is difficult to evaluate the goodness of the results from these visualizations. It will help the paper if the renderings are improved.\n\n4) The discussion in Section 4.3 are good. A similar discussion will be good for “Quantitative Comparison” paragraph that compares LIMR with LASR and ViSER. This is missing and the “Quantitative Comparison” paragraph is simply a plain run of Table1. Some intuition on the performance comparison is missing, perhaps due to the page limit?\n\n5) Having just CD as the eval metric for reconstruction task does not provide a full picture. Additional evaluation metrics such as EMD, F-Score, Normal Consistency and even IoU will provide a good picture of the quality of reconstructions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)