Multiview Human Body Reconstruction from Uncalibrated Cameras
Authors: Zhixuan Yu
Linguang Zhang
Yuanlu Xu
Chengcheng Tang
LUAN TRAN
Cem Keskin
Hyun Soo Park
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D human body reconstruction
Multiview
Uncalibrated
Fusion
Dense keypoints

Abstract: We present a new method to reconstruct 3D human body pose and shape by fusing visual features from multiview images captured by uncalibrated cameras. Existing multiview approaches often use spatial camera calibration (intrinsic and extrinsic parameters) to geometrically align and fuse visual features. Despite remarkable performances, the requirement of camera calibration restricted their applicability to real-world scenarios, e.g., reconstruction from social videos with wide-baseline cameras. We address this challenge by leveraging the commonly observed human body as a semantic calibration target, which eliminates the requirement of camera calibration. Specifically, we map per-pixel image features to a canonical body surface coordinate system agnostic to views and poses using dense keypoints (correspondences). This feature mapping allows us to semantically, instead of geometrically, align and fuse visual features from multiview images. We learn a self-attention mechanism to reason about the confidence of visual features across and within views. With fused visual features, a regressor is learned to predict the parameters of a body model. We demonstrate that our calibration-free multiview fusion method reliably reconstructs 3D body pose and shape, outperforming state-of-the-art single view methods with post-hoc multiview fusion, particularly in the presence of non-trivial occlusion, and showing comparable accuracy to multiview methods that require calibration.

Multiview Human Body Reconstruction from Uncalibrated Cameras
Zhixuan Yu University of Minnesota yu000064@umn.
Linguang Zhang Meta Reality Labs 1 inguangemeta. com
Yuanlu Xu Meta Reality Labs Research yuanluxu@metar com
Chengcheng Tang
Luan Tran
Meta Reality Labs chengcheng. tang@meta. com
Meta Reality Labs ran uan 00meta.........mmmmmmmm.mm...... com
Cem Keskin Meta Reality Labs wwnikkssii@mmta.cc. com
Hyun Soo Park University of Minnesota hspark umn edu
Abstract
We present new method to reconstruct 3D human body pose and shape by fusing visual features from multiview images captured by uncalibrated cameras. Existing multiview approaches often use spatial camera calibration (intrinsic and extrinsic parameters) to geometrically align and fuse visual features. Despite remarkable performances, the requirement of camera calibration restricted their applicability to real-world scenarios, e.g., reconstruction from social videos with wide-baseline cameras. We address this challenge by leveraging the commonly observed human body semantic calibration target, which eliminates the requirement camera calibration. Specifically, we map per-pixel image features to a canonical body surface coordinate system agnostic to views and poses using dense keypoints (correspondences). This feature mapping allows us to semantically. instead geometrically, align and fuse visual features from multiview images. We learn a self-attention mechanism to reason about the confidence of visual features across and within views. With fused visual features, a regressor learned to predict the parameters of body model. We demonstrate that our calibration free multiview fusion method reliably reconstructs 3D body pose and shape, outperforming state- of-the-art single view methods with post-hoc multiview fusion, particularly in the presence of non- soivialvial occlusion, and showing comparable accuracy to multiview methods that require calibration.
1
Introduction
Cameras are an integral part of our lives for us to capture and share priceless moments. In particular, social videos voluntarily captured by multiple viewers watching the scene, e.g., friends recording a street busker simultaneously, provide new form popular content for visual communication in social media.
These videos possess two properties. (1) The videos are, by nature, multiview from social members. which provide redundant yet distinctive visual information to model the 3D body geometry of target subject. For instance, the busker's 's right elbow is occluded by his torso in one view but visible from another viewpoint shown in Figure [[) (2). This self-occlusion can be reasoned by consolidating multiview information. (2) These videos are, general, not spatially calibrated (intrinsic and extrinsic parameters). There exists neither a calibration pattern nor common visual features to register 31)
36th Conference Neural Information Processing Systems (NeurIPS 2022).

---

I X
F2
Φ
F
Iz x
Ø
Uncalibrated multiview images
3D body shape and pose
(a) Input and output
(b) Geometric fusion
(c) Semantic fusion
Figure I: (a) We present a new method to reconstruct the 3D human body shape and pose by fusing visual features from uncalibrated multiview images. All results in this manuscript are generated before blurring the face of the subject. (b) Fusing visual features (F) from multiview images require camera calibration to align features. (c) Instead, we use body semantics, e.g., dense keypoints, to align features, eliminating the requirement of camera calibration.
cameras. Existing computer 39 49 are therefore not applicable. On the other hand, single view approaches [21 23 47 18 for 3D body reconstruction can be combined in post-hoe way, e.g., taking the average of global visual features, which shows limited performance.
camera poses (e.g., structure from motion) because of
vision solutions leveraging multiview calibration [14 [6
13
In this paper, we propose a novel method to reconstruct 3D human pose and shape from uncalibrated multiview images. We use the commonly observed human body as semantic calibration targel to align visual features across views: the dense keypoints (correspondences) [[]] detected in an image provide an injective mapping from image coordinates to the canonical body surface coordinates regardless of views and poses, which allows semantically aligning and then, fusing visual features from multiview images.
We integrate the following desired properties learn visual features for reconstructing 3D body pose and shape. (1) View independence: Without camera calibration, in order to learn and fuse visual features independent of views, we need to transport them to a shared space. To achieve this, we semantically, instead of geometrically. transfer and align visual features based on dense keypoint mapping to the body surface. Such a unified coordinate system makes visual feature extraction agnostic to views and poses. (2) Locality: We hypothesize that local visual feature for each pixel can describe the fine-gra aen-ggrined relationship between human body parts around the pixel. In conjunction with global features that describe a holistic pose, we learn local features from multiview images, jointly. (3) Confidence: Views are not equally informative and complement each other in inferring different body parts. We design an attention mechanism to reason about the confidence of visual features, both globally across multiple views and locally within each view. Across multiple input images, higher global confidence score is assigned to view generating more accurate reconstruction, e.g., less occluded one. Within each view, higher local confidence scores are assigned for body parts that are easier to infer, e.g., a clearly visible limb.
We design an end-to-end network that learns to reconstruct 3D body geometry from multiview images. It takes as input, a set of multiview images and extracts per-pixel features and their confidences. We use dense keypoint mapping to semantically align the features in the canonical coordinate system defined by the human body surface. These aligned features from multiview images are fused by using the predicted confidences, which form unified per-vertex features. We concatenate those fused per-vertex features together with image feature for learning body model parameters.
We apply our method to reconstruct the 3D human body from uncalibrated multiview images. Our method can effectively fuse visual features from multiview images, outperforming single view approaches with post-hoe fusions, particular, for self-occlusion, and showing comparable perfor- mance compared to multiview approaches that use calibrated images. We demonstrate our work in realworld scenarios such social videos where camera calibration is challenging.
Contributions This paper makes three major technical contributions. (1) We propose a novel multiview method for human pose and shape reconstruction that scales up to an arbitrary number of uncalibrated camera views (including single view), guided by dense keypoints. (2) We introduce a novel self-attention based multiview feature fusion method that takes uncertainties across and within different views into consideration. (3) We achieve state-of-the-an results among calibration-fres 31) human body pose and shape reconstruction approaches.

---

2
Related Work
To handle the challenge due to occlusion, single-view- sisgle-view-bbseee based methods infer the occluded parts in a data-driven manner, while multiview approaches can fuse visual features from different views.
Single View 3D Human Reconstruction Reconstructing occluded 3D content from a single view image is a geometrically ill-posed problem. With some assumptions about scenes, it is possible to reason about the occluded geometry. For instance, for humans, a low dimensional parametric model such as SMPL [32] can be used to reconstruct shape and pose by matching 2D sparse landmarks 99
and semantic segmentation masks [26]. On the other hand, learning-based approaches 30 22
directly regress the model parameters from an image [21], or semantic representations such as 2D keypoint heatmaps and silhouettes [37], part segmentation [35]. and dense correspondences [47
Such learning-based approaches rely on datasets with 3D ground truth [3] 20 33 50 to learn the
relationship between an image and the shape and pose, which is prone to overfitting when applying to novel scene image. To address the limited 3D ground truth data, self-supervised learning can be used by generating photorealistic data 233 and by enforcing consistency for predictions from multiple views [6 48 To enhance the expressibility of the model, the parametric representation is relaxed by incorporating a volumetric representation (45 [7]. point cloud (8) or vertex locations of a pre-defined mesh [24 [52. Compared with single view reconstruction, our approach is tailored for multiview feature fusion that significantly improves the performance.
Multiview Fusion for 3D Human Reconstruction When camera calibration (intrinsics and ex- trinsics) is available, single view reconstruction methods can be extend to multiview settings. For instance, SMPLify [3] extended to reconstruct a 3D body geometry in an unified coordinate system where its validity can be measured by projecting onto multiview images, i.e., the reprojection error of the 2D keypoints and silhouette is minimized to learn a geometrically coherent model (28) Multiview geometry offers a geometric way to fuse multiview features for self-ocelusion reasoning 57). For instance, visual features [13 or predictions [50] can be transferred to other views through epipolar
lines, inverse projection [6 44
or a latent view-invariant space 39
When the camera calibration is not available, the learning- based approaches can be used to fuse multiview features. For example, fusion module can be learned to warp a feature map from one view to align with that of another view [38, [46). which does not rely calibration but requires learning a large number parameters to model the cross view geometry. Besides, uncalibrated multiview images can be used in a sequential manner. For example, a recurrent framework can be used to progressively refine initial single view prediction of camera and human model parameters, view by view and stage by stage 229]. The requirement of synchronization between cameras can relaxed by learning probabilistic distribution of the model parameters 42). Unlike existing work, we fuse multiview images without calibration using the observed human body semantic calibration target. The features are transported to the canonical body surface where a self-attention module is used to combine multiview features effectively.
Dense Keypoints for 3D Human Reconstruction Dense keypoint estimation, e.g. DensePose establishes correspondences between image pixels and the canonical body surface coordinate system. This provides a strong cue to infer 3D body pose and shape from a single image that alleviates the self-occlusion problem [40]. Due to such desirable property, single view approaches employ dense
correspondence for supervision 10 52
or as an additional cue [53 47). Further, it can be used to
reconstruct 3D body pose and shape from multiview images [27] 51). Inspired by DecoMR (52), we
leverage dense correspondences to re- arrange visual features from different images into canonical body surface, which allows us to effectively fuse multiview features. Note that DecoMR (52) is a single view method that transfers features to a 2D canonical body surface coordinate and regress a discrete view-specific vertex location map; while ours transfer features from arbitrary number of views to a continuous 3D canonical body mesh for fusion and regressing a view-indepdent body model.
3 Method
We present a novel method that reconstructs 3D human body pose and shape from arbitrary number of uncalibrated multiview images. We leverage dense keypoints map visual features from multiview images to a common canonical coordinate system, which allows effectively fusing them for 3D reconstruction. Figure shows the overview of our pipeline.
3

---

Glohal stiention
Global Renure fision
MEP
Global festure
MIP
Local
Local attertion
Imaage
Fused Econure
Image
Regressor
ft
(30
Lacal feature
UV mep
Image
(n) Encoder
Multiview fusion decoder
neural architecture for the feature encoder fE that outputs visual features for 3D reconstruction. The global feature computed from the bottleneck feature, and its global attention is obtained by transforming the global feature through fully connected layers. The encoder also output per per-pixel (local) features and their attention that mapped to body vertex coordinates via dense keypoints (UV map). (b) We combine the global and local features from the multiview images using their attention. The fused local features and global features are concatenated and feed into MLP regressor for predicting 3D shape and pose.
Figure 2: (a) We design
3.1 Human Body as Semantic Calibration Target
Given an image E [0, 1] #x3 containing person with width W and height H, we aim to recover the person's s 3D body pose and shape represented as mesh vertices = {z, E R by estimating SMPL model [32) parameters e.
This problem in general can be formulated as a composition of an image encoder and a mesh decoder:
F fT(
Z FF 0D),
where fE( denotes image feature encoder that extracts feature maps F from the input image Z. and fD(::0)) denotes the mesh decoder that takes a set of the image features to reconstruct the 3D human pose and shape.
Existing multiview body pose and shape reconstruction methods extend single view methods by leveraging camera calibration to fuse multiview image information [3] 16]. contrast, we leverage the observed human pose as a semantic cue to fuse visual features from multiview images. Consider discrete mapping d(x, I)) that maps pixel in image Z(x) to vertex on the 3D body surface, z,, This offers a new way to align visual features in common vertex coordinate system across views:
F; = fE(x, I; 0F), where i= (xx, I),
(2)
and the encoder fe(+) learns the corresponding image features F, for a certain body vertex z, We assume the 3D body vertex index are consistent across people, views, and poses.
Figure (b..c) illustrates the comparison between geometric fusion that relies on camera calibration to align visual features from multiview images via epipolar line [I3 triangulation [16]. In contrast, our approach uses mappings from image points to 3D body vertices, ((x, Z), semantically align features without camera calibration.
3.2 Local Feature Registration
key challenge learning visual features from Equation (2) lies in the non-differentiabiliiy of the index map Φ. The domain (pixel coordinates x) is not continuous, and any x that does not map to integer index is undefined. We address this challenge by relaxing the discrete domain, i.e., representing the feature F; using set of continuous vicinity features from a continuous mapping.
Using a dense (continuous) keypoint map (x, that maps pixel coordinates x e R² to continuous UV coordinates u E IR², define the feature of the i-th body vertex F, as weighted sum of all features nearby in the UV space (Figure 3):
F; En EueeN Wy WySE(E ((u,),I:0E)
Eujers 16

---

Image
UV
Figure 3: We approximate the discrete feature mapping & using continuous mapping v, i.e. the feature in the i-th vertex is approximated by the weighted average of the nearby features, which enables differentiable feature learning.
Here, NG is a set of neighboring locations of the i-th vertex for which the inverse are well-defined pixel coordinates; w is the weight for the visual feature corresponding to UV coordinates uj. Given the location of the i-th vertex in the UV coordinate system Vs, its neighborhood is defined as:
N₁ (u, d(u,, vs) <,, + - (u,) E Az),
where d(.,)) measures the geodesic distance between two points in the 3D body surface and Az is the lattice of the input image. We further define the weight by applying an RBF kernel on the geodesic distance, i..,
d(u,, v.)² 20
Wi = exp
(5)
where Wy is the weight for the j-th neighbor uy, and controls the effective range of the neighbors. To avoid numerical instability, in practice, we assign zero for features when EujEN, Wy < 10-6.
3.3 Multiview Feature Fusion via Self-attention
Depending on the camera configuration and the human pose, some views can be more informative for reconstructing certain body area than the others which suffer from occlusion or depth ambiguity. We address this by self-attention mechanism that learns assign a weight for each view different body areas.
With Equation 2 visual features from multiview images can be aligned with respect to the 3D body surface indices, which allows to fuse the image features from multiview images:
F
Σ
a F
where F is the i-th fused image feature and of the weight indicating the feature confidence for the e-th camera view, respectively.
We represent the feature confidence of self attention, and it can be learned in unsupervised way:
ag (u) = [A((((( (.))::;)))):.))
where (u) the attention score the UV coordinates u from the e-th view, and fa (x;0s)) ) the learned function that predicts the attention given the set of features from the c-th view, respectively.
Similar to Equation (3) we represent the feature confidence as weighted sum of attention scores from neighboring locations:
exp(X
where B Enjens (f, exp(3t)'
(8)
En E.N, wy
and wi is the weight for the j-th neighbor, uy. in the UV-coordinate system, defined in Equation (5). We apply the softmax operation to normalize feature confidence across views. Aigurrtillustraa the transferred learned attention values the body mesh. Overall, the learned attention scores
consistent with our intuition about contributions to body regions from different camera views.
While per-pixel features are useful for capturing fine-grained geometry, the holistic shape such as pose can be better represented by a global feature. In conjunction with per-pixel features, we incorporate
5

---

F
B
F
B
Figure 4: We use a self-attention mechanism to fuse the local and global features across views. The intensity in the heatmaps (right) indicates the attention predicted by the images (left). "F" denotes the front view and "B" denotes the back view. The attention expected to highly correlated with the visible regions of the body because the features can be confidently predicted from the encoder. The reconstructed 3D mesh overlays the input image.
global features. To fuse global features across views, we utilize a self-attention mechanism similar to local feature fusion with minor modifications. Instead of learning single score for each view, we learn an attention score for each channel of the global feature from each view and then normalize to be used for combining global features across views.
3.4 Loss and Network Design
Given the neural network output θ and subsequent Z, we learn our model in an end-to-end manner by minimizing the following loss:
C= Σ | 1--
Σ ||
P 3D 11
AK Σ [[I[]Z] - 112
(9)
@eDy
PaD
PaDEDR
where three terms are the losses for model parameters, 3D joint error, and 2D joint error, respectively, and Ap and ÅR balance the loss magnitudes. Dv. Dr. and Dx are their datasets, i.e., we use multiple datasets to improve generalizability.
E R PaD E R3M and Pan E RRM are the ground truth model parameters, 3D vertices, 3D joint positions, and 2D joint positions, with N and M denoting the number of vertices and joints, respectively. J E R 3M x3N is the pre-defined 3D joint regression matrix that linearly maps the 3DD vertices to the 3D joints
32 and IT(.) is the operation of camera projection that projects the 3D joints to 2D joints in the image plane. Le measures the Euclidean error between the ground truth model parameters and predicted ones, Lp measures the 3D error of the joints, i.e., JZ the M joint locations, and LK measures the 2D projection of the joint locations.
Note that neither the local feature registration nor the multiview fusion assumes fixed camera configuration or a fixed number of views. We fully leverage this advantage in our training pipeline: at each step, we randomly choose subset of views from each data sample to train the model. As result, we are able mix multiview datasets and single view datasets to train a model that can handle the variable number of views.
3.5 Implementation Details
We design the encoder fE(:;0g) as ResNet-50 backbone [12], that takes a 224 224x3 image as an input and outputs global feature vector with 256 dimensions and a 56 local feature map with 256 dimensions. The local attention function (- AA) is modeled by a sequence layers, while the per-channel attention for global feature is modeled by a fully connected layer. Each local feature vector associated with vertex of down-sampled SMPL mesh (24) goes through three-layer MLP and reduces its dimension from 256 to [54], and then we concatenate all 431 of them to form 2155-ddmmnnsoos feature vector. This aggregated local feature vector is then concatenated with the global feature vector and fed into a MLP regressor similar to that in HMR 21
convolutional
We use a pre-trained DensePose model for vh which partitions the whole body surface 24 different parts where UV map is defined for each part. In practice. d(u v) in Equation (4) and Equation (5) is set infinity if and v from different parts. Another important detail is that for vertex that corresponds to multiple 2D UV coordinates, we take the minimum when computing the geodesic distance, i.e., replacing d(u,, v:) with min( [d(u,, v)))) ) in Equation (4) and Equation (5). where v is the k-th 2D UV coordinates that the same i-th vertex corresponds. In fact, this serves as approximation of using v. We set 0.05 and 2.33 1

---

4 Experiments
We validate our calibration-free free multiview fusion approach on multiple datasets varying from indoor to outdoor, controlled to in-the-wild environments. We refer the reader to check additional results, experiments and implementation details in the supplementary material.
4.1 Datasets
Human3.6N [15] is a large-scale multiview dataset with ground truth 3D human pose annotation.
We follow the standard training/testing split: using subject S1, S5, S6, S7 and S8 for training, and subject S9 and S11 for testing. We reconstruct the ground truth 3D human mesh in the format of
the SMPL (32] model by applying MoSh 33 the sparse 3D MoCap marker following previous
works 21
52 51]
UP-3D 26 is an in-the-wild single view dataset containing 8,515 images annotated with ground
truth 2D keypoints and 3D body mesh. 3D body mesh is obtained by fitting the SMPL [32] model
images from the Leeds Sports Pose [19]. MPII Human Pose [ and FashionPose [S] datasets. We
use the standard training split [24 52
MARCOnI [6] is multiview dataset including both indoor and outdoor images for evaluating marker-less motion capture methods. This dataset contains 12 real-world sequences captured by 3-16
cameras with varying modalities from 1-2 subjects each in different scenes, e.g., Soccer, Kickbox, Juggling, etc. We use this dataset to qualitatively evaluate our method and test the generalizability of the model.
VBR 22 is multiview dataset that include challenging social videos captured by multiple members in a scene. The baseline between cameras are wide. It contains three synchronized multiview sequences including Juggler, Magician, and Rothman. We use this dataset qualitatively evaluate our method and test the generalizability of the model.
Social Video are new multiview dataset we collected for qualitative study. It features multiple persons performing dynamic group activities, such as playing basketball and name tag ripping, which introduces frequent occlusion and disocclusion. Up to subjects are captured from 4 hand-held cameras in 6 sequences. We run an off-the-shelf 2D human pose detection and tracking tool [7] with manual correction and multiview association to obtain ground truth per-person multiview tracking data.
Note that we run an off-the-shelf dtructure-frommmotiee system (COLMAP 41] to solve the camera parameters for multiview datasets above. However, expected, for almost all cases the bundle adjustment fails to converge and it reports that no good initial image pair was found, i.e., lack of correspondences.
4.2 Metrics
Since the proposed method reconstructs the 3D body shape and pose (mesh) and uses the local features for recovering fine details, we evaluate of the method using Mean Per- Vertex Position Error (MPVPE) and Mean Per-Joint Position Error (MPJPE), both after Procrustes Analysis (PA). MPJPE-PA primarily evaluates the pose estimation accuracy of the model, whereas MPVPE-PA evaluates the accuracy of the reconstructed body shape.
4.3 Comparison with State-of-the-art Methods
Baselines SMPLify 3 optimize SMPL model parameters by fitting to 2D keypoints. HMR 21 reconstructs the 3D body shape and pose from monocular image by estimating the SMPL model parameters in an iterative manner. GraphCMR [24] leverages a graph convolutional network to first reconstruct the non-parametric mesh which is then refined by fitting SMPL model parame- ters. SPIN (23) further improves the accuracy by combining iterative fitting and regression into self-improving loop. DecoMR 122 Pose2Mesh 44, 12IMeshnet (34), Tessoranffrmmer 30 and PyMAF [54] are several more recent monocular methods. Multiview SPIN and LVS [43] are two methods that leverage multiview calibration for estimating SMPL model parameters reference.
Onlly authors from University of Minnesota downloaded and accessed to Human3.6M dataset. Authors from Meta Reality Labs don't 't have access to it. Social Videos dataset is collected at University of Minnesota, not Meta Reality Labs.
7

---

Multiview
Single
Type
Calibration-free
MPJPE-PA
MPVPE.PA
SMP sty
Mono
NIA
NIA
82.3
NIA
HMR 21
Mono
57.8 + 10.7
67.7 = 15.4
56.8
05.3
Graplx
CM
P4
Mono
30.9
9.1
19.1
13.4
561.
56.8
SPIN 23
Mono
44.5 4 7.9
51.5
11.
41.1
49.3
42.0
121Msshmet
Mono
N/A
NIA
41.1
NIA
Transfomel
+0
Mono
N/A
N/A
36.7
NIA
PyMAF [4
Mono
N/A
N/A
10.5
NIA
MV-SPIN
65]
Multi
35.1
N/A
N/A
NIA
LVS
43
Mulai
32.5
NIA
N/A
NIA
Linne
Multi
57
G0.1
69.2
ProHMR
Mulá
34.5
NIA
41.2
NIA
Our
Multi
33.0
31.4
11.6
46.4
Table Comparison between the proposed method and existing single view and multiview methods on Human3.6 6M dataset. For each method, we specify the supported type of input and whether camera calibration is required. We evaluate the performance of each method on both multiview and single view reconstruction. Since multiview reconstruction is not supported by single view methods, we report the mean and standard deviation of the error after evaluating the method on each of the input views. In addition, we report the multiview methods (MV-SPIN 43] and LVS 43) that use camera calibration for fusion as a reference, i.e., it is expected provide the performance upper bound of our calibration-free method. MPJPE-PA and MPVPE-PA are reported in millimeter.
To our best knowledge, Liang require known camera calibration.
29 and ProHMR 25 are the only multiview methods that does not
Multiview Reconstruction We reconstruct 3D body shape and pose using multiview images from Human3.6k dataset as summarized Table middle three columns. For single view methods, we
observe large variations in accuracy due to the varying viewing angle. Multiview methods overcome this limitation by fusing information from multiple views and producing a unified output. Figure 5
shows such a comparison between ours and SPIN [23] at continuous frames of a Human3.6M [15]
sequence under MPVPE-PA. Compared to Liang [29] and ProHMR 255. our method outperforms in both MPJPE-PA and MPVPE-PA. We also report the performance of the multiview methods (MV-SPIN [43] and LVS (43) that rely on the camera calibration as a reference, i.c., it is expected to provide the performance upper bound of our calibration-fiee method.
Single View Reconstruction Our method can be applied to an arbitrary number of cameras, including monocular camera, without retraining the model. Applying our method on single view input, however, does not benefit from the attention-based multiview feature fusion. Evaluation results can be found in Table last two columns. Even without specifically targeting for the single view use case, the results produced by our method comparable with existing ones on MPJPE-PA (except MeshTrenssee Transformer (30)) that use a transformer network as backbone) while outperform others on MPVPE-PA. Further, fusion method is complementary to the single view reconstruction approaches where we can extract the global and local features with minor modification. We expect that a stronger performance can be achieved in the multiview reconstruction if a stronger baseline can be used.
4.4 Ablation study
For all the following experiments, we train the model using the Human3.6M and UP-3D datasets, and evaluate the model on the test split of Human3.66 M...
Effectiveness of Local Features We compare our model with the variant that is trained with the global features only. The features across views are fused with the self-attention mechanism. Table 2 shows that adding local features effectively improves 3D reconstruction accuracy.
Effectiveness of using Self-attention To demonstrate the effectiveness of our attention based fusion, compare ours with other alternative fusion methods: average pooling and max pooling. Comparison in Table demonstrates that the model trained with our self-attention fusion outperforms the other fusion methods by a large margin.
Benefit of using More Views To investigate the benefit of using more views for 3D reconstruction, we evaluate the performance of our model with varying number of views, randomly selected from all four views. Note that re-training the model is not necessary since our approach allows taking an arbitrary number of views as input. The results summarized in Table Using more views helps

---

200
Single-view
175
MPVPE-PA (unitt min)
Single-view 1
Variant
MPIPE-PA
MPVPE-PA
Single-view
local feature
37.7
12.0
125
Single-view 3
Average fusion
35.4
37.8
fusion
35.2
37.5
100
Mulli-view
AIL Jusson (ours)
33.0
34.4
views)
34.2
75
Att. fusion (2 views)
37.3
40.2
Att. fusion view)
44.1
50.1
50
AlL fusion (1 view, fromt)
41.5
46.1
AIL Jusson (1 view, back)
46.5
53.2
25
Table 2:
We compare different variants of the proposed method verify the de- sign decisions. All the variants are trained the union of Human3.6M and UP-3D and evaluated on Human3.6N (use all four views unless specified), MPJPE-PA and MPVPE-PA are reported in millimeter.
100
300
400
500
Frame index
Figure 5: We compare ours that infer jointly from mul- tiple views with SPIN [23] infer each view indepen- dently on a sequence from Human3.6M [15]. Our multi-
view results are equivalent to or better than single-view
result from the best view for almost all frames.
improve the performance. Also, compared frontal views (camera and 3), back views (camera O and 2) that are more prone to self-occlusion benefit the most from information fused from other views.
4.5 Qualitative Results
We apply our method to reconstruct 3D body shape and pose using multiview images and show the qualitative comparisons images from Human3. 6M (15). VBR 22. MARCOnl 6 and self-collected
Social Videos dataset against SPIN [23] in Figure
4.6
Failure Cases
In Figure 7 we show qualitative results of failure cases including (1) failure of dense keypoints (example A and B) and (2) under severe occlusion or crowded (example C).
Summary and Future Work
We present a novel 3D human body reconstruction method that uses multiview images without calibration. Instead of relying geometric camera calibration, human subject as a semantic calibration target that can align the visual features from multiview images. The visual features are learned in the 3D body surface coordinate, established by the dense keypoint mapping, that is agnostic views and poses. We reason about self-occlusion via the confidence of the aligned features, which allows us to effectively fuse the features from multiview images. The fused features are fed into a MLP regressor to regress model parameters. Experiments show that calibration-free method is able to effectively leverage multiview information in principled way, outperforming state-of satte-of-thh-an single view approaches with a post-hoc multiview fusion. Our approach is readily applicable to real world scenarios, including 3D reconstruction from social videos.
method leverages the observed
Our reliance on the dense keypoint estimation can double-edged sword. While it establishes the dense correspondences without camera calibration, the erroneous estimation of the dense keypoints may lead to misalignment of the local features, resulting in suboptimal 3D reconstruction. Our atten- tion mechanism can mitigate fusing erroneous features if the majority of dense keypoint estimation from multiview images are correct. In practice, we found that the dense keypoint estimation (II) is highly reliable, which makes method applicable in-the-wild social videos.
We believe camera calibration, if available, provides additional point-to-ling correspondences, which can be used to not only validate, but also refine the point-to-point correspondences our feature fusion method relies on. Exploring a combination of both, along with developing robust techniques to perform in- in-the-wild multiview calibration, is a promising future direction. Another potential future work to extend our method to leverage temporal information and produce more temporally coherent predictions.

---

V1
(a) Qualitative comparison with SPIN
23
Human3.66 3.6M
16
B
D
(b) Qualitative results on Unstructured VBR (A, C). and MARCOa (D).
View (V1)
Single view ree. froud V1
Single viewaree. frem VI
Multiview reconstractien
Mulpsiew recunstraction
View (V2)
Singleview From V2
Single vievoree, Froiu/
Multiview reconstruction
Multsstw groputraccion
(c) Qualitative comparison with SPIN 23
Social Videos dataset.
Figure 6: (a) While reconstruction results of both may have similar level of alignment with image after reprojection, 3D reconstructions of single-view method are not consistent across views due to bias of viewpoint. "V" stands for "view". "S" stands for "Single-view reconstruction [23]". "M" stands for "Multiview reconstruction"
ours, "G" stands for "Ground truth". (b, c) We reconstruct 3D body shape and pose from uncalibrated multiview images of characteristic social videos in diverse scenes, including Unstructured VBR, MARCOnI, and our Social Videos. While single view reconstruction produces inconsistent poses depending on views, ours produces a consistent pose invariant to views as leveraging multiview information jointly.
Figure 7: Failure cases. Dense keypoint estimation is missing for left leg in example A and right arm for example B for both views. In example C, subject rendered with reconstruction rendered red is under severe occlusion or crowded for both views.
Acknowledgement
This work was partially supported by NSF NRI 2022894 and NSF CAREER 1846031.
10

---

References
[1] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR, 2014.
[2] L.. Ballan, G. J. Brostow, Puwein, and M. Pollefeys. Unstructured video-based rendering: Interactive exploration of casually captured videos. In SIGGRAPH, 2010.
F. Bogo, A. Kanazawa, C. Lassner, P. Gehler, Romero, and M. J. Black. Keep it smpl: Automatic estimation of 3d human pose and shape from single image. In ECCV, 2016.
[4] H. Choi, G. Moon, and K. M. Lee. Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from 2d human pose. In ECCV, 2020.
[5] Dantone, Gall, C. Leistner, and Van Gool. Human pose estimation using body parts dependent joint regressors. CVPR, 2013.
[6] A. Elhayck, E. Aguiar, A. Jain, J. Tompson, L. Pishchulin, M. Andriluka, C. Bregler, B. Schiele, and C. Theobalt. Efficient convnet-based marker-less motion capture in general scenes with low number of cameras. In CVPR, 2015.
[7] H.-S. Fang, S. Xie, Y.W.. Tai, and C. Lu. RMPE: Regional multi-person person pose estimation. In ICCV, 2017.
[8] V. Gabeur, J.-S. Franco, Martin, C. Schmid, and G. Rogez. Moulding humans: Non-parametrie 3d human shape estimation from single images. ICCV, 2019.
[9] P. Guan, A. Weiss, A. O. Balan, and M. Black. Estimating human shape and pose from a single image. In ICCV, 2009.
[10] R. A. Guler and I. Kokkinos. Holopose: Holistic 3d human reconstruction in- w--the-wild. In CVPR, 2019.
[11] R. A. Güilrr, N. Neverova, and I. Kokkinos. Densepose: Dense human pose estimation in the wild. In CVPR, 2018.
[12] K. He. X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[13] Y. He, R. Yan, K. Fragkiadaki, and S.-I. Yu. Epipolar transformers. In CVPR, 2020.
[14] Y. Huang, F. Bogo, C. Lassner, A. Kanazawa, P. V. Gehler, J. Romero, I. Akhter, and M. J. Black. Towards accurate marker-less human shape and pose estimation over time. In 3DV, 2017.
[15] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. Large scale datasets and predictive methods for 3d human sensing in natural environments. PAMI, 2013.
[16] K. Iskakov, E. Burkov, Lempitsky, and Y. Malkov. Learnable triangulation of human pose. In ICCV, 2019.
[17] A. S. Jackson, C. Manafas, and G. Tzimiropoulos. 3d human body reconstruction from single image via volumetric regression. In ECCV, 2018.
[18] Y. Jafarian H. S. Park. Learning high fidelity depths dressed humans by watching social media dance videos. In CVPR, 2021.
[19] S. Johnson and M. Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC, 2010.
[20] H. Joo, H. Liu, L. Tan, L. Gui, B. Nabbe, I. Matthews, T. Kanade, S. Nobuhara, and Y. Sheikh. Panoptic studio: A massively multiview system for social motion capture. In ICCV, 2015.
[21] A. Kanazawa, CVPR, 2018.
Black, D. Jacobs, and Malik. End-to-end recovery human shape and pose. In
[22] M. Kocabas, C.-H. P. Huang, O. Hilliges, and M. J. Black. Pare: Part attention regressor for 3d human body estimation. In ICCV, 2021.
[23] N. Kolotouros, G. Pavlakos, M. Black, and K. Daniilidis, Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In ICCV, 2019.
[24] N. Kolotouros, Pavlakos, and K. Daniilidis. Convolutional mesh regression for image human shape reconstruction. CVPR, 2019.
[25] N. Kolotouros, G. Pavlakos, D. Jayaraman, and K. Daniilidis. Probabilistic modeling for human mesh recovery. In ICCV, 2021.

---

[26] C. Lassner, J. Romero, M. Kiefel, F. Bogo, M. J. Black, and P. V. Gehler. Unite the people: Closing the loop between 3d and 2d human representations. In CVPR, 2017.
[27] X. Li, Y. Liu, Jon, Q. Dai, and Y. Sheikh. Capture dense: Markerless motion capture meets dense pose estimation. arXiv, 2018.
[28] Z. Li, M. Oskarsson, and A. Heyden. 3d human pose and shape estimation through collaborative learning and multi-view model-fitting. In WACV, 2021.
[29] J. Liang and M. C. Lin. Shape-aware human pose and shape reconstruction using multi-view images. In ICCV, 2019.
[30] K. Lin, L. Wang, and Liu. End-to-end human pose and mesh reconstruction with transformers, In CVPR, 2021.
[31] M. Loper. N. Mahmood, and M. J. Black. Mosh: SIGGRAPH, 2014.
Motion and shape capture from sparse markers.
[32] Loper, Mahmood, Romero, Pons-Moll, and M. J. Black. Smpl: A skinned multi-person linear model. SIGGRAPH, 2015.
[33] D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Somnychnnko, W. Xu, and C. Theobalt. Monocular 3d human pose estimation in the wild using improved enn supervision. In 3DV, 2017.
[34] G. Moon and K. M. Lee. 121-meshnet: Image-to-lixx -ixelll prediction network for accurate 3d human pose and mesh estimation from single rgb image. In ECCV, 2020.
[35] M. Omran, C. Lassner, G. Pons-Moll, Gchler, and B. Schiele. Neural body fitting: Unifying deep lcarning and model based human pose and shape estimation. In 3DV, 2018
[36] G. Pavlakos. N. Kolotouros, and K. Daniilidis. Texturepose: Supervising human mesh estimation with texture consistency. In ICCV, 2019.
[37] G. Pavlakos, Zhu, X. Zhou, and Daniilidis. Learning estimate human pose and shape from single color image. In CVPR, 2018.
[38] H. Qiu, C. Wang. J. Wang, N. Wang, and W. Zeng. Cross view fusion for 3d human pose estimation. In ICCV, 2019.
[39] E. Remelli, S. Han, S. Honari, P. Fua, and R. Wang. Lightweight multi-view 3d pose estimation through camera- dinetaa-ise representation. In CVPR, 2020.
[40] Y. Rong, Z. Liu, C. Li. K. Cao, and C. C. Loy. Delving deep into hybrid annotations for 3d human recovery in the wild. In ICCV, 2019.
[41] J. L. Schonberger and J.-M. Frahm. fimcture-from-mmttio revisited. In CVPR, 2016.
[42] A. Sengupta, I. Budvytis, and R. Cipolla. Prohahilistic 3d human shape and pose estimation from multiple unconstrained images in the wild. In CVPR, 2021.
[43] S. Shin and E. Halilaj. Multi-view human pose and shape estimation using learnable volumetric aggregation. arXiv, 2020.
[44] H. Tu, C. Wang, and Zeng. Voxelpose: Towards multi-camera 3d human pose estimation in wild environment. ECCV, 2020.
[45] G. Varol, D. Ceylan, B. Russell, J. Yang, E. Yumer, I. Laptev, and C. Schmid. Bodynet: Volumetric inference of 3d human body shapes. In ECCV, 2018.
[46] R. Xie, C. Wang, and Y. Wang. Metafuse: A pre-trained fusion model for human pose estimation. In CVPR, 2020.
[47] Y. Xu, S.-C. Zhu, and T. Tung. Denserac: Joint 3d pose and shape estimation by dense render-and-comparc. In ICCV, 2019.
[48] Y. Yao, Y. Jafarian, and H. S. Park. Monet: Multiview semi-supervised keypoint detection via epipolar divergence. In ICCV, 2019.
[49] Z. Yu. S. Horvath, A. Delazio, Wang, R. Almasi, R. Klatzky, Galeotti, and Stetten. Palmsight: an assistive technology helping the blind locate and grasp objects. CMU Technical Report, 2016.
[50] Z. Yu, J. Yoon, I. K. Lee, P. Venkatesh, J. Park, J. Yu, and H. S. Park. Humbi: A large multiview dataset of human body expressions. In CVPR, 2020.
12

---

[51] Z. Yu, H. Yu, L. Sha. S. Ganguly, and H. S. Park. Semi-supervised dense keypoints using unlabeled multiview images. In NIPS, 2021.
[52] W. Zeng, W. Ouyang, P. Luo, W. Liu, and X. Wang. 3d human mesh regression with dense correspondence. In CVPR, 2020.
[53] Zhang, Cao, Lu, Ouyang, Sun. Learning 3d human shape and pose from dense body parts. PAMI, 2020.
[54] H. Zhang, Y. Tian, X. Zhou, W. Ouyang, Y. Liu, L. Wang, and Z. Sun. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In ICCV, 2021.
[55] Y. Zhang, Z. Li, L. An, M. Li, T. Yu, and Y. Liu. Lightweight multi-person total motion capture using sparse multi-view cameras. In ICCV, 2021.
[56] Z. Zhang, C. Wang, Qiu. W. Qin, W. Zeng. Adafuse: Adaptive multiview fusion for accurate human pose estimation in wild. IJCV, 2020.
[57] Y. Zheng, R. Shao, Y. Zhang. T. Yu, Z. Zheng, Q. Dai, and Y. Liu. Deepmulticap: Performance capture of multiple characters using sparse multiview cameras, In ICCV, 2021.
13


Reviews:
[{"review_id": "ke0AIzoSAEa", "reviewer": "Reviewer_XdJs", "summary": "The authors present a technique to estimate the underlying pose and shape of a body from multiview images without requiring calibration information. The technique uses the human body as a calibration target and performs fusion of local and global features via a self-attention mechanism. The authors demonstrate the efficacy of the approach on standard benchmarks. ", "questions": "1. Did the authors try to replace the self-attention mechanism with an explicit calibration based fusion mechanism? It would be useful to understand the similarities and differences of the self-attention with such an explicit fusion mechanism beyond average and max pooling discussed in the paper. \n2. Please perform a quantitative evaluation on the social videos dataset as the the method is best suited for such a setting. This would entail calculating the relative pose between multiview images for comparison to the other methods, but this is a critical contribution and the wider audience would benefit from an analysis on \"in-the-wild\" dataset.\n3. How does the method compare in run-time to other comparable methods?Did the authors experiment with bulkier or leaner verions of the architecture? A variant that runs in real-time would be especially useful for social videos setting. \n4. Why havent the authors evaluated against [34] which seems to be solving the problem in a more relaxed setting? The authors should show some examples with single camera videos which are multiview over time, albiet with pose changes. The behavior of the method in such a setting, along with associated failure modes would be useful to understand from an application perspective.  ", "limitations": "The authors haven't discussed the failure modes of the approach adequately. The social videos dataset presents a good opportunity to qualitatively demonstrate when the method fails as mentioned in the limitations section (e.g. under severe occlusion). The authors haven't discussed the performance of the method on single camera videos which have greater practical appeal. Also the possibility to futher improve the approach with calibration by adapting the self-attention mechanism warrants some discussion to guide future work. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of using the body as a calibration target is novel. \n2. The method shows comparable accuracy to methods requiring calibration which is not always available. \n2. The authors have collected a new dataset (Social Datasets) to qualitatively evaluate the proposed approach. \n\nWeaknesses:\n1. The approach is not sufficiently motivated. Such a multiview approach applies to a multi-camera rig setting, and calculating the calibration is straight-forward. The authors should provide a more compelling discussion on why the method is of practical importance?  \n2. The method stands to be most useful in the context of the Social videos dataset. However, no quantitative evaluations exist on the same. \n3. The assumption in line \"141\" of \"We assume the 3D body vertex index i are consistent across people, views, and poses\" is a strong assumption and forms the basis of the approach. Some quantitative analysis or commentary on this assumption will be useful for wider adoption. Some failure cases should also be shown. \n4. The self-attention mechansim proposed in the paper serves to remove dependence on calibration and is the main contribution of the paper. The authors could do a better analysis of the usefulness of self-attention mechanism. \n5. The architecture has limited novelty. For e.g., the self-attention mechanism could be reformulated as a transformer based multi-headed attention module. It would be useful to understand if such a formulation is beneficial. \n6. The complexity and run-time of the method is not discussed. \n\nPost rebuttal:\nThe authors have adequately responded to my concerns. I would have liked to visually see the failure cases. I am slightly increasing the rating of the paper based on addressing of my concerns. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "d3Du8pShSoB", "reviewer": "Reviewer_Lrh9", "summary": "The paper describes human pose and reconstruction with a multi-view method that can handle up to an arbitrary number of uncalibrated camera views. \nThe proposed method maps visual features from different views to a canonical body surface to reconstruct a fused 3D SMPL parametric model. \nDepending on each view, a learnable parameter assigns confidence to the fused features, playing the role of attention. \n\nHuman3.6M, UP-3D, MARCOnI and VBR are the datasets taken into account for collecting different views. Experiments are done solely on Human3.6M,  with the P1 protocol. Concerning the considered competitors, the proposed method shows the best performance on MPVPE-PA, both for single and multi-view approaches.  Qualitative results are also shown on Social Videos dataset.\n", "questions": "There is minimal information about the experiments. Essential facts are the number of images used and whether different runs have been made with a changing number of images for the multi-view testing. \nSimilarly, for training, the method can randomly choose images to construct a model from multi-views, what kind of protocol is used to choose images, and how many samples are collected, especially in training. \n\nThe authors suggest that multi-view is relevant for solving occlusions, and in their Conclusions the authors write that there are cases in which it cannot be solved. A comparison with other methods, such as “PARE: Part Attention Regressor for 3D Human Body Estimation” (which uses a weak perspective camera), and a discussion on the conditions under which the method fails to solve occlusions is relevant for a complete understanding of the method.  \n\nThe proposed loss uses both 3D and 2D information and the SMPL parameters \\Theta; given that recently, many methods do not require paired 2D-3D data and only use weak – supervision, would the proposed multi-view implant be able to extend to unpaired 2d-3d data, using the multi-view?\nWhat about runtime and number of parameters:  is the model competitive in terms of computation time and size? \n", "limitations": "Limitations are discussed in particular about self-occlusions", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The merit of the paper is to devise a multiview method using only key features and mapping them to a canonical model, which should, in principle, solve self-occlusion problems.\nThe paper is well structured, and the images are good, though it is not so well written.\n\nA relevant weakness is a choice of experimenting only on Human3.6M.  \nWhy not also use 3DPW, which consists of 35515 RGB images of 7 subjects with paired ground-truth SMPL parameters.\nMany recent contributions use 3DPW for testing. \n\nFurthermore, to compare with the single view methods, the authors report the mean and standard deviation of the error of each method tested on several individual inputs. \nThe comparison might be unfair since a single view method does not take advantage of several views. \n\nDespite multi-views being claimed to be relevant for occlusions, there are no ablation studies on this issue.\nRecent contributions such as “End-to-End Human Pose and Mesh Reconstruction with Transformers” and “Probabilistic Modeling for Human Mesh Recovery” have better results on single images than the authors. The authors should compare their method with the most recent works.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KI-LAWnyLDx", "reviewer": "Reviewer_w6Ne", "summary": "This paper presents a calibration-free method to reconstruct 3D human body pose and shape from multiview images. The proposed method is motivated by the observations that the human body can be leveraged as a semantic calibration target. Experimental results validate the effectiveness of the proposed method. Both qualitative and quantitative results are promising.", "questions": "- The supplementary material provides a version using GCN to regress mesh vertices directly, but the quantitative results are counterintuitive as the single-view methods with non-parametric representations typically outperform the parametric counterparts numerically. But it makes sense that there are artifacts in the reconstructed meshes of the non-parametric solutions, as shown in Figure 3 of the supplementary material. It is also recommended to add ablation experiments that use the GCN mesh decoder when camera parameters are available.\n\n- The related work could be improved to be more comprehensive.  It is recommended to include [A] in the related work, as it is a recent state-of-the-art multiview reconstruction method when camera calibration is available.\n\n[A] Lightweight multi-person total motion capture using sparse multi-view cameras. ICCV 2021.\n", "limitations": "Please see the comments in Weaknesses and Questions.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Generally, this paper is well-motivated, well-written, and technically sound. It is quite smooth to read this paper.\n\nOverall, this paper presents an interesting solution for uncalibrated multi-view human pose and shape estimation.\n\n+ The motivation of leveraging the human body as a semantic calibration target is appreciated.\n\n+ The approximation of the discrete feature mapping via continuous mapping is interesting, which is the key to achieving differentiable feature learning. It is reminiscent of the existing differentiable rendering approaches.\n\nSome concerns regarding the current version of the paper:\n\n- Some part of the technical details is not very clear. It is confusing to me how to concatenate the per-pixel features with the global features?\n\n- How about the proposed method when the camera calibration is available? The continuous mapping may also contribute to better fusion when the camera parameters are known. Adding such a setting in the ablation study should strengthen the paper.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "F1lEVxCkdv1", "reviewer": "Reviewer_bCFQ", "summary": "This paper presents a new method for 3D human pose reconstruction give multiple (or a single) image without the need for camera parameters. The method is based on doing dense pose estimation, then predicting local and global features for how salient parts of an image are from a specific view. Finally it regresses parameters of a SMPL model which parameterizes the human body.\n\n------------\nPost rebuttal: Changing score from 6 to 7. I think the authors can improve the presentation of the paper and with that I would rate it a 7. I like the approach and design along with the motivation of using the human body as the semantic target. ", "questions": "What's the reason why the final parameters were regressed instead of using an optimization based approach like SMPLify? It should be possible to do that without a calibration (optimize the 2D pixel loss from the output to the dense keypoint mapping) and it would ensure the output matches up with the visible images as much as possible.\n\nIt seems like the main proposed applications is for videos, and there is a large body of work on 3D pose from video. Is there a reason the authors chose to specifically focus on multi-view images without using temporal information?", "limitations": "Limitations are well addressed in the final section.\n\nIf the authors can address questions about the presentation and clarity, and include that in a revised version, then I would consider increasing my review score given that the contribution and method are interesting.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe intuition behind the paper makes a lot of sense especially the way it was presented in terms of global features and local features with an attention map. I particularly liked the experiments section. There were detailed comparisons to other methods, and the ablation study did a good job of justifying the different design decisions. I would have liked to see a comparison with SMPLify in the table as well, but a number of other single view baselins were presented.\n\nWeaknesses:\nThe presentation of the paper could be improved, especially the methods section. For example, figure 4 which shows the full method overview should appear near the top, especially before figure 3 and 2. Seeing the system overview is important before diving into the specific pieces. Figure 4 should also include $F_A$ so we can see where those weights are. It also was not clear how the weights for $F_A$ are learned. Eq. 7 has two unknowns: $a_c$ which is the attention score and $F_A$ which is the weights of the attention module. Do we have access to the ground truth attention score during training in order to learn $F_A$?\n\nThe other main weakness is that the results shown in the video still contain considerable artifacts especially in the way the hand moves. Compared to other methods for related 3D reconstruction tasks, such as Nerf based methods, the artifacts are very noticeable. The results are better than the single channel baseline, but could still be better especially given the number of camera views.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CSW468pGcMO", "reviewer": "Reviewer_RhpY", "summary": "A 3D human body reconstruction method using multiview images without calibration is presented. The method leverages semantic calibration to align visual features learned from multiview images by dense keypoints detection.  ", "questions": "- It is not clear how the global features and local features are used in this method. Could you add more details?\n- The idea of leveraging multiple datasets with different GT is very attractive. Could the authors provide details on how the training is performed?\n ", "limitations": "- Qualitative results are shown to validate the multiview methods advantageous over single view in terms of consistency across views. This point is obvious. Thus, more qualitative results might be more valuable. \n\n- Semantic calibration idea is sound for multiview reconstruction. If the idea could be extended to single view, or video-based reconstruction, that will be more practically useful. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of using semantic calibration for human body reconstruction is sound and novel. Leveraging 2D/3D keypoints detection as well as multiple datasets make it an interesting approach. \n- Main concepts and motivations are well and clearly delivered.\n\nWeaknesses:\n- The application of multiview reconstruction requires that sequences are temporally aligned. Comparing the single view methods, this might limit the use cases of the proposed method.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept