Direct Multi-view Multi-person 3D  Pose Estimation
Authors: Tao Wang
Jianfeng Zhang
Yujun Cai
Shuicheng YAN
Jiashi Feng
Venue: NeurIPS 2021 Poster (2021)
Keywords: Human Pose Estimation
3D
Multi-view
Transformer
End-to-End
Direct
Camera Ray

Abstract: We present Multi-view Pose transformer (MvP) for estimating multi-person 3D poses from multi-view images. Instead of estimating 3D joint locations from costly volumetric representation or reconstructing the per-person 3D pose from multiple detected 2D poses as in previous methods, MvP directly regresses the multi-person 3D  poses  in  a  clean  and  efficient  way,  without  relying  on  intermediate  tasks. Specifically, MvP represents skeleton joints as learnable query embeddings and let them progressively attend to and reason over the multi-view information from the input images to directly regress the actual 3D joint locations. To improve the accuracy of such a simple pipeline, MvP presents a hierarchical scheme to concisely represent query embeddings of multi-person skeleton joints and introduces an input-dependent query adaptation approach. Further, MvP designs a novel geometrically guided attention mechanism, called projective attention, to more precisely fuse the cross-view information for each joint. MvP also introduces a RayConv operation to integrate the view-dependent camera geometry into the feature representations for augmenting the projective attention.  We show experimentally that our MvP model outperforms the state-of-the-art methods on several benchmarks while being much more efficient. Notably, it achieves 92.3% AP25 on the challenging Panoptic dataset, improving upon the previous best approach [35] by 9.8%. MvP is general and also extendable to recovering human mesh represented by the SMPL model, thus useful for modeling multi-person body shapes. Code and models are available at https://github.com/sail-sg/mvp.

Direct Multi-view Multi-person 3D Pose Estimation
Tao Wang's .2 Jianfeng Zhang²* Yujun Shuicheng Yan Jiashi Jiashi Feng',
Sea Al Lab "National University of Singapore,
twangnh@gmail. com, zhangj ianfjjinnnngeee nus. edu, {caiyj yansc, fngg @eeee
Abstract
We present Multi-view Pose transformer (MvP) for estimating multi-person 3D poses from multi-view images. Instead of estimating 3D joint locations from costly volumetric representation or reconstructing the per-person 3D pose from multiple detected 2D poses as previous methods, MvP directly regresses the multi-person 3D poses in clean and efficient way. without relying intermediate tasks. Specifically, MvP represents skeleton joints as learnable query embeddings and let them progressively attend to and reason over the multi-view information from the imput images directly regress the actual 3D joint locations. To improve the accuracy of such simple pipeline, MvP presents a hierarchical scheme to concisely represent query embeddings multi-person skeleton joints and introduces input- dependent query adaptation approach. Further, MvP designs novel geometrically guided attention mechanism, called projective attention, to more precisely fuse the cross-view information for each joint. MvP also introduces a RayConv operation to integrate the view-dependent camera geometry into the feature representations for augmenting the projective attention. We show experimentally that MvP model outperforms the state-of-the-at methods on several benchmarks while being much more efficient. Notably, it achieves 92.3% AP25 on the challenging Panoptic dataset, improving upon the previous best approach [40] by 9.8%. MvP is general and also extendable to recovering human mesh represented by the SMPL model, multi-person
thus useful for modeling multi person body shapes. Code and models are available at https: //github. com/ sailssllssssssss aal-sg/mvp.
Introduction
Multi-view multi-person 3D pose estimation aims to localize 3D skeleton joints for each person instance in scene from multi-view camera inputs. It is fundamental task that benefits many real-world applications (such as surveillance, sportscast, gaming and mixed reality) and is mainly tackled by reconstruction-based [6. 14, 4] and volumetric [40] approaches in previous literature, as shown in Fig. (a) and (b). The former first estimates 2D poses in each view independently and then aggregates them and reconstructs their 3D counterparts via triangulation or a 3D pictorial structure model. The volumetric approach [40] builds 3D feature volume through heatmap estimation and 2D-to-3D un- projection at first, based on which instance localization and 3D pose estimation are performed for each person instance individually. Though with notable accuracy, the above paradigms are inefficient due to highly relying on those intermediate tasks. Moreover, they estimate pose for each person separately, making the computation cost grow linearly with the number of persons.
Targeted more simplified and efficient pipeline, were wondering if it is possible directly regress 3D poses from multi-view images without relying on any intermediate task? Though con- ceptually attractive, adopting such direct mapping paradigm is highly non-trivial it remains
"Equal Contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

unclear how to perform skeleton joints detection and association for multiple persons within a single stage. In this work, we address these challenges by developing a novel Multi-view Pose transformer (MvP) model which significantly simplifies the multi-person 3D pose estimation. Specifically, MvP represents each skeleton joint as learnable positional embedding, named joint query, which is fed into the model and mapped into final 3D pose estimation directly (Fig. 1 (c)), via a specifically designed attention mechanism to fuse multi-view information and globally reason over the joint predictions to assign them to the corresponding person instances. We develop novel hierarchical query embedding scheme to represent the multi-person joint queries. It shares joint embedding across different persons and introduces person-level query embedding to help the model in learning both person-level and joint-level priors. Benefiting from exploiting the person-joint relation, the model can more accurately localize the 3D joints. Further, propose to update the joint queries with input-dependent scene-level information (i.e., globally pooled image features from multi-view inputs) such that the learnt joint queries can adapt to the target scene with better generalization performance.
To effectively fuse the multi-view information, we propose a geometrically-guidee projective attention mechanism. Instead of applying full attention to densely aggregate features across spaces and views, it projects the estimated 3D joint into 2D anchor points for different views, and then selectively fuses the multi-view local features near to these anchors to precisely refine the 3D joint location. we propose to encode the camera rays into the multi-view feature representations via a novel RayCony operation to integrate multi-view positional information into the projective attention. In this way, the strong multi-view geometrical priors can be exploited by projective attention to obtain more accurate 3D pose estimation.
Comprehensive experiments on 3D pose benchmarks Panoptic [19], as well as Shelf and Campus [1] demonstrate our MyP works very well. Notably, it obtains 92.3% AP25 on the challenging Panoptic dataset, improving upon the previous best approach VoxelPose [40] by 9.8%, while achieving nearly 2x speed up. Moreover, the design ethos of our MyP can be easily extended to more complex tasks-we show that simple body mesh branch with SMPL representation [28] trained on top of a pre-trained MvP can achieve competitively qualitative results.
Our contributions are summarized as follows: 1) We strive for simplicity in addressing the challenging multi-view qulti-person 3D pose estimation problem by casting it as direct regression problem and accordingly develop a novel Multi-view Pose transformer (MvP) model, which achieves state-of- the-art results on the challenging Panoptic benchmark. 2) Different from query embedding designs in most transformer models, we propose more tailored and concise hierarchical joint query embedding scheme to enable the model to effectively encode person-joint relation. Additionally, we mitigate the commonly faced generalization issue by a simple query adaptation strategy. 3) We propose a novel projective attention module along with RayConv operation for fusing multi-view information effectively, which we believe are also inspiring for model designs in other multi-view 3D tasks.
2 Related Works
3D Human Pose Estimation 3D pose estimation from monocular inputs [29, 30, 49, 35, 38, 31, 46, 10, 47] is an ill-posed problem as multiple 3D predictions may result in the same 2D projection. To alleviate such projective ambiguitics, multi-view methods have been explored. Research works on single-person scenes use either multi-view geometry [11] for feature fusion [36, 13] and trian- gulation [16, 37J, or pictorial structure models for fast and robust 3D pose reconstruction [34, 36], achieving promising results. However, it is more challenging as we progress towards multi-person scenes. Current approaches mainly exploit a multi-stage pipeline for multi-person tasks, including reconstruction-bes based [6, 4, 14, 21, 26] and volumetric [40] paradigms. Despite their notable accuracy, these methods suffer expensive computation cost from the intermediate tasks, such as cross-view matching and heatmap baok-projection. Moreover, the total computation cost grows linearly with the number of persons in the scene, making them hardly scalable for larger scenes. Different from all previous approaches that rely on multi-stage pipeline with computation redundancy, our method views multi-person 3D pose estimation as a direct regression problem based on a novel Multi-view Pose transformer model, enables an intermediate task-free single stage solution.
Attention and Transformers Driven by the recent success in natural language fields, there have been growing interests in exploring the Transformers for computer vision tasks, such as image recognition [8] and generation [18], as well as more complicated object detection [3, 51] and video

---

M
A
po0mg
pergenn Regression
Multi-view
Person Proposal Genenntion
Croes-view Matching
Joint Query
Quary
2D Un teeppeecccioo
festure
Teature
2D
Pose
Detectio
Detestor
CNN
CNN
CNN
View
Mulli-siew Inguts
(a)
(b)
(c)
Figure 1: Difference between our method and others for multi-view multi-person 3D pose estimation. Existing methods adopt complex multi-stage pipelines that are either (a) iiconstructionbbseee or (b) volumetric representation based, which incur heavy computation burden. (c) Our method solves this task as direct regression problem without relying on any intermediate task by a novel Multi-view Pose Transformer, and largely simplifies the pipeline and boosts the efficiency.
instance segmentation [42]. However, multi-person 3D pose estimation has not been explored along this direction. In this study, we propose novel Multi-view Pose Transformer architecture with joint query embedding scheme and a projective attention module to regress 3D skeleton joints from multi-view images directly, delivering a simplified and effective pipeline.
3 Multi-view Pose Transformer (MvP)
To build direct multi-person 3D pose estimation framework from multi-view images, we introduce novel Multi-view Pose transformer (MvP). MvP takes in the multi-view feature representations, and transforms them into groups of 3D joint locations directly (Fig. 2 (a)), delivering multi multi-person person
3D pose results, with the following carefully designed query embedding and attention schemes for detecting and grouping the skeleton joints.
3.1 Joint Query Embedding Scheme
Inspired by transformers [41], MvP represents each skeleton joint learnable positional embedding, which is fed into the transformer decoder and mapped into final 3D joint location by jointly attending to other joints and the multi-view information (Fig. (a)). The learnt embeddings encode prior knowledge about the skeleton joints and we name them as joint queries. MvP develops the following concise query embedding scheme.
Hierarchical Query Embeddings The most straightforward way for designing joint query embed- dings is to maintain learnable query vector for each joint per person. However, we empirically find this scheme does not work well, likely because such a naive strategy cannot share the joint-level knowledge between different persons.
To tackle this problem, we develop a hierarchical query embedding scheme explicitly encode the person-join relation for better generalization to different scenes. The hierarchical embedding offers joint- information sharing across different persons and reduces the learnable parameters, helping the model to learn useful knowledge from the training data, and thus generalize better. Concretely, instead of using the set of independent joint queries (4 C C we employ a set person level queries (h, J.V C RRC and set of joint level queries (1,) C RC to represent different persons and different skeleton joints, where C denotes the feature dimension,
is the number of persons, J is the number of joints per person, and M N... Then the query of joint j of person

---

Joint
Confidence Location
otkle
3D Pose Estimasion
View
Dacoler Layer
T
View
Transformer Decoder
Decoder Layer
View N
Ray N
Decoder
Queries
Projected Anchor Point
3D 2D Projection
Deformable Point
Roy Operation
(a) Pipeline overview
(b) Projective attention
Figure 2: (a) Overview of the proposed MvP model. Upon the multi-view image features from several convolution layers, it deploys a transformer decoder with a stack of decoder layers to map the input joint queries and the multi-view features to 3D poses directly. (b) The projective attention of MvP projects 3D skeleton joints to anchor points (the green dots) on different views and samples deformable points (the red dots) surrounding these anchors aggregate local contextual features via learned weights (the brighter color density means larger weights).
can be hierarchically formulated as
g = h 1-
With such hierarchical embedding scheme, the number of learnable query embedding parameters reduced from NJC' to (N + J)C.
Input-dependent Query Adaptation In the above, the Iearned joint query embeddings are shared for all the imput images, independent of their contents, and thus may not generalize well on the novel target data. To address this limitation, we propose to augment the joint queries with input-dependent scene-level information in both model training and deployment, such that the learnt joint queries can be adaptive to the target data and generalize better. Concretely, augment the above joint queries with globally pooled feature vector E RC from the multi-viee view image feature representations:
q h +I.
Here = Concat(Pool Po. ())) where Zy denotes image feature from v-th view and V is the total number of camera views; Concat and Pool denote concatenation and pooling operations, and 4 a learnable linear weight.
3.2 Projective Attention for Multi-view Feature Fusion
It is crucial to aggregate complementary multi-view information to transform the joint embeddings into accurate 3D joint locations. We consider the dot product attention mechanism of transformers [41] to fuse the multi-view image features. However, naively applying such dot product attention densely over all spatial locations and camera views will incur enormous computation cost. Moreover, such dense attention is difficult to optimize and delivers poor performance empirically since it does not exploit any 3D geometric knowledge.
Therefore, we propose ccometrically-guidee multi-view projective attention scheme, named pro- jective attention. The core idea is take the 2D projection of the estimated 3D joint location the anchor point in each view, and only fuse the local features near those projected 2D locations from different views. Motivated by the deformable convolution [5, 50], we adopt an adaptive deformable sampling strategy to gather the localized context information in each camera view, as shown in Fig. 2 (b). Other local attention operations [48. 44, 43] can also be adopted as an alternative. Formally, given joint query feature q and 3D joint position the projective attention is defined as
PAttention(g.. (q. (Z. ) Concat (fncat((f, f,, W..
(3)
where
Σ akk) Z, (T(y,, C.) Ap(k))
Here the view-specific feature f,, is obtained by aggregating features from K discrete offsetted sampling points from anchor point p 11(y, C.), located by projecting the current 3D joint

---

location to 2D, where II: R + R2² denotes perspective projection [11] and C, the corresponding camera parameters. W and w are learnable línear weights. The attention weight and the offset to the projected anchor point Ap are estimated from the fusion of query feature q and the view-dependent feature at the projected anchor point Z. (p). í.e., a = Softmax Ap = (q + (P)) where W and w are learnable linear weights. If the projected location and the offset are fractional, we use bilinear interpolation to obtain the corresponding feature Z, (p) or Z. (p+ + Ap()).
(p)) and
The projective attention incorporates two geometrical cues, i.e., the corresponding 2D spatial locations across views from the 3D 2D projection and the deformed neighborhood of the anchors from the learned offsets to gather view-adaptive contextual information. Unlike naive attention where the query feature densely interacts with the multi-view key features across all the spatial locations, the projective attention is more selective for the interaction between the query and each view view-only -only the features from locations near to the projected anchors are aggregated, and thus is much more efficient.
Encoding Multi-view Positional Information with RayConv The positional encoding [41] is an important component of the transformer, which provides positional information of the input sequence. However, a simple per-view 2D positional encoding scheme cannot encode the multi-view geometrical information. To tackle this limitation, we propose to encode the camera ray directions that represent positional information in 3D space into the multi-view feature representations. Concretely, the camera ray direction R... generated with the view-specific camera parameters, concatenated channel-wisely to the corresponding image feature representation Z.. Then a standard convolution is applied to obtain the updated feature representation Z.,, with the view-dependent geometric information:
Z. Cony(Concat( (Z... R.)).
(4)
We name the operation as RayConv. With it, the obtained feature representation Ż. is used for the projective attention by replacing Z, in Eqn. (3).
Such drop-in replacement introduces negligible computation, while injecting strong multi-view geometrical prior to augment the projective attention scheme, thus helping more precisely predict the refined 3D joint position.
3.3 Architecture
Our overall architecture (Fig. 2 (a)) pleasantly simple. It adopts a convolution neural network, designed for 2D pose estimation [45], to obtain high-resolution image features \ from multi- view inputs [I,]o=I- The features are then fed into the transformer decoder consisting of multiple decoder layers to predict the 3D joint locations. Each layer conducts self-attention to perform pair-wise interaction between all the joints from all the persons in the scene; a projective attention to selectively gather the complementary multi-view information; and a feed-forward regression to predict the 3D joint positions and their confidence scores. Specifically, the transformer decoder applies a multi- layer progressive regression scheme, i.e., each decoder layer outputs 3D joint offsets to refine the input 3D joint positions from previous layer.
Extending to Body Mesh Recovery MvP learns skeleton joints feature representations and is extendable to recovering human mesh with a parametric body mesh model [28]. Specifically, after average pooling on the joint features into per-person feature, feed-forward network is used to predict the corresponding body mesh represented by the parametric SMPL model [28]. Similar to the joint location prediction, the SMPL parameters follow multi-layer progressive regression scheme.
3.4 Training
MyP infers a fixed set of M joint locations for N different persons, where M NJ. The main training challenge is how to associate the skeleton joints correctly for different person instances. Unlike the post-hoe grouping of detected skeleton joints as in bottom-up pose estimation methods [32, 24], MyP learns to directly predict the multi-joint 3D human pose in a group-wise fashion as shown in Fig. (a). This is achieved by a grouped matching strategy during model training.
Grouped Matching Given the predicted joint positions (y,, 1 C R and associated confidence scores {sm JM Jm group every consecutive J-joint predictions into per-person pose estimation
5

---

(Y,)== C R and average their corresponding confidence scores to obtain the per-person confidence scores {Pa J. The same grouping strategy is used during inference.
The ground truth set Y of 3D poses of different person instances is smaller than the prediction set of size N, which is padded to size N with empty element Ø. Then find bipartite matching between the prediction set and the ground truth set by searching for permutation of E NN that achieves the lowest matching cost:
ô arg min
Σ Ematch (Y,, Y vo(n))-
(5)
TENN
We consider both the regressed 3D joint position and confidence score for the matching cost:
Lmatch YY,, Y Ya(n)) -; + (Y,,, n² Ya(n))
(6)
where Y; * Ø, and L1 computes the L1 loss error. Following [3, 39], we employ the Hungarian algorithm [25] to compute the optimal assignment ā with the above matching cost.
Objective Function
We compute the Hungarian loss with the obtained optimal assignment à
Hungarian (Y", YY,))
Σ Ccont n Po(n)) I Yn Yago)]
Here Lcont and Pose are losses for confidence score and pose regression, respectively. À balances the two loss terms. We use focal loss [27] for confidence prediction which adaptively balances the positive and negative samples. For pose regression, compute I, loss for 3D joints and their projected 2D joints in different views.
To learn multi- llyer progressive regression, the above matching and loss applied for each decoder layer. The total loss is thus Liotal Σt= Huungariaan where Hungarian denotes loss of the 1-th decoder layer and L is the number of decoder layers. When extending MvP to body mesh recovery, we apply L loss for 3D joints from the SMPL. model and their 2D projections, as well adversarial loss following HMR [22, 17, 47] due to lack of GT SMPL parameters.
4 Experiments
In this section, we aim to answer following questions. 1) Can MyP provide both efficient and accurate multi-person 3D pose estimation? 2) How does the proposed attention mechanism help multi-view multi-person skeleton joints information fusing? 3) How does each individual design choice affect model performance? To this end, we conduct extensive experiments on several benchmark datasets.
Datasets Panoptic [20] large-scale benchmark with 3D skeleton joint annotations. It captures daily social activities in indoor environment. We conduct extensive experiments Panoptic to evaluate and analyze our approach. Following VoxelPose [40], we use the same data sequences except 160906 band3' in the training set due to broken images. Unless otherwise stated, we use five HD cameras (3, 6, 12, 13, 23) in our experiments. All results reported in the experiments follow the same data setup. We Average Precision (AP) and Recall [40], well as Mean Per Joint Position Error (MPJPE) as evaluation metrics. Shelf and Campus [1] are two multi multi-person datasets capturing indoor and outdoor environments, respectively. We split them into training and testing sets following [1, 6, 40J. We report Percentage of Correct Parts (PCP) for these two datasets.
Implementation Details Following VoxelPose [40], we adopt pose estimation model [45] build upon ResNet-50 [12] for multi-view image features extraction. Unless otherwise stated, we use
Table 1: Result on the Panoptic dataset. MvP is more accurate and faster than VoxelPose.
Methods
AP2s
APso
AP100
AP100
Recallasoo
MPJPE[mm]
Time[ms]
VoxelPose [40]
84.0
96.4
97.5
97.8
98.1
17.8
320
MvP (Ours)
92.3
96.6
97.5
97.7
98.2
15.8
170

---

stack of six transformer decoder layers. The model is trained for 40 epochs, with the Adam optimizer of learning rate 10 During inference, confidence threshold of 0.1 is used filter out redundant predictions. Please refer to supplementary for more implementation details.
4.1 Main Results
Panoptic We first evaluate MvP model the challenging Panoptic dataset and compare with the state-of-the-art VoxelPose model [40].
MvP-100 persons (185ms)
1156
As shown in Table 1, Our MvP achieves 92.3
Inference time (ms)
1000
MvP-20 persons (174ms)
MvP-10 persons (170ms) VoxclPase
AP25. improving upon VoxelPose by 9.8%, and
achieves much lower MPJPE (15.8 v.s 17.8).
600
GPU Geforoe RIX 2080 II
GRU: 11-600D 1,0091
465
Moreover, MvP only requires 170ms process multi-view input, about 2x faster than Voxel- Pose These results demonstrate both accuracy and efficiency advantages of MvP from estimat-
400
352
416
300
309
260
200
ing 3D poses of multiple persons in direct regression paradigm. To further demonstrate efficiency of MvP, we compare its inference time with VoxelPose's when processing differ- ent numbers of person instances. As shown in Fig. 3, the inference time of VoxelPose grows linearly with the number of persons the scene due to the per-person regression paradigm. In contrast, MvP keeps constant inference time no matter how many instances in the scene. Notably, it takes only 185ms for MvP process scenes with 100 person instances (the blue line), demonstrating its great potential to handle crowded scenarios.
5
15 20
Number of person instances
Figure 3: Inference time versus the number of per- son instances. Benefiting from its direct inference framework, MvP maintains almost constant infer- ence time regardless of the number of persons.
Shelf' and Campus We further compare our MvP with state-of-thed of-thhrart approaches on the Shelf and Campus datasets. The reconstruction-based methods [2, 9, 6] use 3D pictorial model [2, 6] condi- tional random field [9] within a multi-stage paradigm; and the volumetric approach VoxelPose [40] highly relies on computationally intensive intermediate tasks. shown in Table 2, our MvP achieves the best performance in all the actors on the Shelf dataset. Moreover, it obtains a comparable result on the Campus dataset VoxelPose [40] without relying on any intermediate task. These results further confirm the effectiveness of MvP for estimating 3D poses of multiple persons directly.
Table 2: Results (in PCP) on Shelf and Campus datasets.
Shelf
Campus
Methods
Actor
Actor
Actor
Average
Actor
Actor
Actor 3 Average
Belagiannis al.
75.3
69.7
87.6
77.5
93.5
75.7
84.4
84.5
Ershadi et al. [9]
93.3
75.9
94.8
88.0
94.2
92.9
84.6
90.6
Dong et al. [6]
98.8
94.1
97.8
96.9
97.6
93.3
98.0
96.3
VoxelPose [40]
99.3
94.1
97.6
97.0
97.6
93.8
98.8
96.7
MvP (Ours)
99.3
95.1
97.8
97.4
98.2
94.1
97.4
96.6
4.2 Visualization
3D Pose and Body Mesh Estimation We visualize some 3D pose estimations of MvP on the challenging Panoptic dataset in Fig. 4. It can observed that MvP is robust large pose deformation (the 1st example) and severe occlusion (the 2nd example), and can achieve geometrically plausible results w.r.t. different viewpoints (the rightmost column). Moreover, MyP is extendable to body mesh recovery and can achieve fairly good reconstruction results (the 2nd and 4th rows). All these results verify both effectiveness and extendability of MyP. Please see supplementary for more examples.
We count averaged per-sample inference time in millisecond on Panoptic test set. For all methods, the time counted GPU GeForce RTX 2080 Ti and CPU Intel 17-6900K @ 3.20GHz.

---

A
Figure 4: Example 3D pose estimations from Panoptic dataset. The left four columns show the multi-view inputs and the corresponding body mesh estimations. The rightmost column shows the estimated 3D poses from two different viewpoints. Best viewed color.
Figure 5: Visualization of projetive attention and self-atee attention on example skeleton joints. The attention weights are obtained with the 4-th decoder layer of trained model. Projective attention (in the cropped image triplets): the green points denote the projected anchor points in each camera view, and the red points denote the offsetted spatial locations, with brighter color for stronger attention. Self-attention (in the 3D skeleton plots): example skeleton joint (green) to all the other skeleton joints (red) in the scene. The color density indicates attention weight. Best viewed in color and 2x zoom.
Attention Mechanism We visualize the projective attention and the self-attention in Fig. 5. Ben- efiting from the 3D-to-2D projection, the projective attention can accurately locate the skeleton joint in each camera view (the green point) based the current estimated 3D joint location. We observe learns to gather adaptive local context information (the red points) with the deformable sampling operation. For instance, when regressing the 3D position of mid-hip (the 1st example), the projective attention selectively attends to informative joints such as the left and right hips as well as thorax, which offers sufficient contextual information for accurate estimation. We also visualize the self-attention, which learns pair-wise interaction between all the skeleton joints in the scene. From the 3D plot in Fig. 5, we can observe a certain skeleton joint mainly attends other joints of the same person instance (more opaque). It also attends to joints from other person instances, but with less attention (more transparent). This phenomenon is reasonable as the skeleton joints of a human body are strongly correlated to each other, e.g., with certain pose priors and bone length.
4.3 Ablation
Importance of RayConv MyP introduces RayConv to encode multi-view geometric information,
i.e., camera ray directions into image feature representations. As shown in Table 3a, if removing RayConv, the performance drops significantly 4.8 decrease in AP25 and increase in MPJPE. This indicates the nuet geometrical information is important for the model to more precisely

---

Table 3:
Ablations Panoptic. In (b), Hier. denotes the hierarchical query embedding scheme, Hier. +ad. means further adding the adaptation strategy. Please see supplement for more ablations.
Query AP2s
AP
MPJPE
Thr.
AP25
AP100
100
MPJPE
RCon AP25
AP100 100
MPJPE
0.0
93.1
98.5
16.3
Per-joint 67.4
84.7
41.2
w/
92.3
97.5
15.8
0.1
92.3
97.5
15.8
Hier.
82.5
93.2
19.5
w/o
87.5
96.2
17.4
0.2
91.1
96.2
15.5
Hier+ad.
92.3
97.5
15.8
(a) The effect of RayCon.. means removing RayConv.
w/o
0.4
89.2
93.7
15.0
(b) Different joint query embedding schemes.
(c) Different confidence threshold during evaluation.
Dec.
AP25
AP100
MPJPE
Cam.
AP25
AP100
MPJPE
2
6.3
92.5
49.6
K
AP2s
AP 100
MPJPE
63.4
95.6
22.8
4.7
61.0
93.8
4
86.8
96.8
17.5
2
37.7
93.0
34.8
88.6
96.3
18.2
5
91.8
97.6
16.2
3
71.8
95.1
21.1
2 89.3
97.5
17.4
6
92.3
97.5
15.8
4
84.1
96.7
19.3
92.3
97.7
15.8
7
92.0
97.5
15.9
5
92.3
97.5
15.8
84.4
91.1
20.3
(d) Number of decoder layers.
(e) Number of camera views.
(I) Number of deformable points K.
localize the skeleton joints in 3D space. Without RayConv, the transformer decoder cannot accurately capture positional information 3D space, resulting in performance drop.
Importance of Hierarchical Query Embedding As shown in Table 3b, compared with the straightforward and unstructured per-joint query embedding scheme, the proposed hierarchical query embedding boosts the performance sharply-14.1 increase in AP25 and 23.4 decrease in MPJPE. Its advantageous performance clearly verifies introducing the person-level queries to collab- orate with the joint-level queries can better exploit human body structural information and improve model better localize the joints. Upon the hierarchical query embedding scheme, adding the query adaptation strategy further improves the performance significantly, reaching AP25 of 92.3 and MPJPE of 15.8. This shows the proposed approach effectively adapts the query embeddings to the target scene and such adaptation is indeed beneficial for the generalization of MvP to novel scenes.
Different Model Designs We also examine effects of varying the following designs of the MvP model to gain better understanding on them.
Confidence Threshold During inference, a confidence threshold is used to to filter out the low- confidence and erroneous pose predictions, and obtain the final result. Adopting higher confidence will select the predictions in a more restrictive way. As shown in Table 3c, a higher confidence threshold brings lower MPJPE as selects more accurate predictions; but also filters out some true positive predictions and thus reduces the average precision.
Number of Decoder Layers Decoder layers are used for refining the pose estimation. Stacking more decoder layers thus gives better performance (Table 3d). For instance, the MPJPE is as high as 49.6 when using only two decoder layers, but it is significantly reduced to 22.8 when using three decoder layers. This clearly justifies the progressive refinement strategy of our MvP model is effective. However the benefit of using more decoder layers diminishes when the number of layers is large enough, implying the model has reached the ceiling of its model capacity.
Number of Camera Views Multi-view inputs provide complementary information to each other which is extremely useful when handling some challenging environment factors in 3D pose estimation like occlusions. We vary the number of camera views examine whether MvP can effectively fuse and leverage multi-view information to continuously improve the pose estimation quality (Table 3e). As expected, with more camera views, the 3D pose estimation accuracy monotonically increases, demonstrating the capacity of MvP in fusing multi-view information.
Number of Deformable Sampling Points Table 3f shows the effect of the number of deformable sampling points K used in the projective attention. With only one deformable point, MvP already achieves respectable result, i.e., 88.6 in AP25 and 17.4 in MPJPE. Using more sampling points

---

further improves the performance, demonstrating the projective attention is effective at aggregating
information from the useful locations. When K 4. the model gives the best result. Further increasing K to 8, the performance starts to drop. It is likely because using too many deformable points introduces redundant information and thus makes the model more difficult to optimize.
5
Conclusion
We introduced a direct and efficient model, named Multi-view Pose transformer (MvP), to address the challenging multi-view multi-person 3D human pose estimation problem. Different from existing methods relying on tedious intermediate tasks, MvP substantially simplifies the pipeline into a direct regression one by carefully designing the transformer-alike model architecture with novel hierarchical joint query embedding scheme and projective attention mechanism. We conducted extensive experiments to verify its superior performance and speed over the well-established baselines.
We empirically found MvP needs sufficient data for model training since learns the 3D geometry implicitly. In the future, will study how to enhance the data-efficiency of MvP by leveraging the strategy like self-supervised pre-training or exploring more advanced approaches. Similar to prior works, we also found MvP suffers from performance drop for cross-camera generalization, that is, generalizing on novel camera views. We will explore approaches like disentangling camera parameters and multi-view feature learning to improve this aspect. Besides, will explore the large-scale applications of MvP and further extend it to other relevant tasks. Thanks to its efficiency, MvP would be scalable to handle very crowded scenes with many persons. Moreover, the framework of MvP is general and thus extensible to other 3D modeling tasks like dense mesh recovery of common objects.
References
[1] Vasileios Belagiannis, Sikandar Amin, Mykhaylo Andriluka. Bernt Schiele, Nassir Navab, and Slobodan Ilie. 3d pictorial structures for multiple human pose estimation. In CVPR, 2014.
[2] Vasileios Belagiannis, Sikandar Amin, Mykhaylo Andriluka, Bernt Schicle, Nassir Navab, and Slobodan
Ilie. 3d pictorial structures revisited: Multiple human pose estimation. IEEE transactions on patter analysis and machine intelligence, 38(10) 1942, 2015.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[4] Chen, Pengfei Guo, Pengfei Li, Gim Hee Lee, and Gregory Chirikjian. Multi Aulti-person pose estimation in crowded scenes based on multi-view geometry. In ECCV, 2020.
[5] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
[6] Junting Dong, Wen Jiang, Qixing Huang. Hujun Bao, and Xiaowei Zhou, Fast and robust multi-person 3d pose estimation from multiple views. In CVPR, 2019.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv, 2020.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition scale. arXiv, 2020.
Sara Ershadi-Nasab Erfan Noury, Shohrch Kasaci, and Esmacil Sanaci. Multiple human 3d pose estimation from multiview images. Multimedia Tools and Applications, 77(12):15533-156 2018.
[10] Kehong Gong, Jianfeng Zhang, and Jiashi Feng. Poseaug: A differentiable pose augmentation framework for 3d human pose estimation. In CVPR, 2021.
[11] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, New York, NY, USA, 2 edition, 2003.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[13] Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar transformers In CVPR, 2020.
[14] Congzhentao Huang. Shuai Jiang. Yang Li, Ziyue Zhang, Jason Traish, Chen Deng, Sam Ferguson, and Richard Yi Da Xu. End-to-enn dynamic matching network for multi multi-view multi person 3d pose estimation. In ECCV, 2020.
multi-person
10

---

[15] Catalin lonescu. Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Trans. on Pattern Analysis and Machine Intelligence, 36(7):1325-1339, 2014.
[16] Karim Iskakov, Egor Burkov, Victor Lempitsky, and Yury Malkov. Learaale triangulation of human pose. In ICCV, 2019.
[17] Wen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis. Coherent reconstruction of multiple humans from single image. In CVPR, 2020.
[18] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two transformers can make one strong gan. arXivv 2021.
[19] Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In ICCV, 2015,
[20] Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, lain Matthews, et al. Panoptic studio: A massively multiview system for social interaction capture. IEEE transactions on pattern analysis and machine intelligence, (1(1):190-20) 2017.
[21]
Abdolrahim Kadkhodamohammadt and Nicolas Padoy. A generalizable approach for multi-view 3d human pose regression. Machine Vision and Applications. 32(1):1-14 14, 2021.
[22] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery of human shape and posc. In CVPR, 2018.
[23] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICCV, 2015.
[24] Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi, Pifpaf: Composite fields for human pose estimation. In CVPR, 2019.
[25] Harold W Kuhn. The hungarian method for assignment problem. Naval research logistics quarterly. 2(1-2):3397, 1955.
[26] Jiahao Lin and Gim Hee Lec. Multi-view multi-person 3d pose estimation with plane sweep stereo. In CVPR, 2021.
[27] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. ICCV, 2017.
[28] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned culti-person linear model, ACM transactions on graphics (TOG), 34(6):1-16, 2015.
[29] Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little. A simple yet effective baseline for 3d human pose estimation. In ICCV, 2017.
[30] Dushyant Mehta. Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. Vnect: Real-time 3d human pose estimation with single rgb camera. ACM Trans. on Graphics, 36(4):44, 2017.
[31] Xuecheng Nie, Jianfeng Zhang, Shuicheng Yan, and Jiashi Feng. Single-stage multi person pose machines. In ICCV, 2019.
[32] George Papandreou, Tyler Zhu, Liang-Chich Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy. Personlah: Person pose estimation and instance segmentation with bottom-up, part-hased, geometric embedding model. ECCV, 2018.
[33] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Leter, Automatic differentiation in pytorch. In NeurlPSw, 2017.
[34] Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Danillidis. Harvesting multiple views for marker-less 3d human pose annotations. In CVPR, 2017.
[35] Alin-Ionut Popa, Mihai Zanfir, and Cristian Sminchisescu. Deep multitask architecture for integrated 2d and 3d human sensing. CVPR. 2017.
[36] Haibo Qiu, Chunyu Wang. Jingdong Wang. Naiyan Wang, and Wenjun Zeng. Cross view fusion for 3d human pose estimation. ICCV, 2019.
[37] Edoardo Remelli, Shangehen Han, Sina Honari, Pascal Fua, and Robert Wang. Lightweight multi-view 3d pose estimation through camera-disentangled representation. In CVPR, 2020.
[38] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In ECCV, 2018
[39] Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. arXiv. 2014.

---

[40] Hanyue Tu, Chunyu Wang, and Wenjun Zeng. Voxelpose: Towards multi-camera 3d human pose estimation in wild environment. In ECCV, 2020.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv, 2017.
[42] Yuqing Wang. Zhaoliang Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. arXiv, 2020.
[43] Felix Wu, Angela Fan, Alexei Bacvski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions, arXiv, 2019.
[44] Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. arXiv, 2020.
[45] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In ECCV, 2018.
[46] Jianfeng Zhang. Xuecheng Nic, and Jiashi Feng. Inference stage optimization for cross- cross-scenario human pose estimation. In NeurlPS, 2020.
[47] Jianfeng Zhang, Dongdong Yu, Jun Hao Liew, Xuecheng Nie, and Jiashi Feng. Body meshes points. In CVPR, 2021.
[48] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun, Exploring self-attention for image recognition. In CVPR, 2020.
[49] Xingyi Zhou, Qixing Huang, Xiao Sun, Xiangyang Xue, and Yichen Wei. Towards human pose estimation in the wild: weakly-supervised approach. In ICCV, 2017.
[50] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. CVPR, 2019.
[51] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end end object detection. In ICCV, 2020.
12


Reviews:
[{"review_id": "twEtZVviH81", "reviewer": "Reviewer_NJn8", "summary": "The paper presents a method for multi-view multi-human 3D pose estimation. The method extracts multi-view feature information using a Transformer architecture and directly regresses the 3D joint locations using the aggregated features. The two proposed technical improvements are 1) representing joints as learnable query embeddings and 2) extracting salient multiview information by aggregating per-view features near the 2D projections of an estimated 3D joint location in each view. The method is shown to improve over prior methods in both accuracy and computational efficiency. The method is evaluated on the Panoptic (Joo et al., 2017) dataset and another multi-person pose estimation dataset (Chen et al., 2020).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The overall method / architecture feels intuitive and conceptually simple, and I appreciate that the paper focuses on inference-time efficiency as a main objective. My comments mainly concern technical details of the method.\n\n1. One key assumption in the joint query embedding scheme is that each person is assigned with a unique learned embedding that's input-independent. How would the method generalize to a new pose estimation target (e.g., previously unseen person)? If my understanding is correct, line 122-130 attempts to address the problem. However, the “global pooled feature” seems to be more suitable for identifying a new scene rather than for generalizing to a new target. I’d like to see a more focused study on how the proposed method generalizes to new targets and whether such per-person embedding negatively impacts the generalization performance.\n\n2. Projective Attention relies on a set of estimated 3D joint position “anchors”. Specifically, where do the y’s in Eq 3. come from? From my understanding, they are crude estimations of the 3D joint positions from each view. How do these 3D position estimations differ from the final predictions? It’d be interesting to show some qualitative examples that demonstrate the importance of aggregating information from multiple views. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- No dedicated discussion for limitations\n- It'd be interesting to see potential societal impact of the work, but it's not essential to my rating considering the nature of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "hfUeAWJeWH", "reviewer": "Reviewer_SX8n", "summary": "In this paper, the authors propose a Multi-view Pose transformer for to address multi-view multi-person 3D human pose estimation problem. Compared with cascaded approaches, the proposed approach substantially simplifies the pipeline into a direct regression with the transformer-alike model architecture, which includes a hierarchical joint query embedding scheme and projective attention mechanism. In the experiments, the authors conduct experiments on CMU-Panoptic dataset and show improvements compared with other competing methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this paper is well written and easy to follow. The authors propose an end-to-end approach for multi-view multi-person 3D pose estimation based on the transformer architecture.\n+ The proposed model uses multi-view feature representations and transforms them into groups of 3D joint locations directly and formulates the problem of multi-view multi-person 3D pose into an one-stage problem, which is quite interesting. \n+ The proposed hierarchical joint query embedding for person-joint relation encoding and projective attention are also quite interesting.\n+ The experiments are conducted on two public benchmarks (CMU Panoptic, Shelf and Campus) and overall quite convincing. The proposed method outperforms the previous state of the art VoxelPose.\n+ Code is attached in supplementary.\n\nI have a few questions and comments for this paper. \n+ How is R_v represented in RayConv? \n+ It would be interesting to see more ablative studies on the input-dependent query adaptation to support the claim that using the globally-pooled feature vector could increase the generalization ability to novel test data. The authors could use some cross-dataset test for validation.\n+ It seems the proposed method always predicts person instances with the full skeleton. How does partial occlusion affect the multi-person 3D pose estimation?\n+ Could the proposed group matching handle some challenging cases, e.g., people hugging each other? The authors are encouraged to discuss more about the limits of the current joint-to-person association strategy.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Overall I feel this paper is well prepared and with good quality. The proposed multi-view pose transformer is quite interesting and could bring some new insights for the people in related field. The authors are encouraged to address my questions above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "hA6GfV1gtGA", "reviewer": "Reviewer_vMiR", "summary": "The paper uses transformers to solve the multi-view multi-person 3d pose estimation problems.  To improve the inference speed, they use the new query, to make the feature more scene-aware they use a global pooled and concatenated feature. To make the feature more geometrical aware they use view-direction as another feature. Better results/ faster speed  are shown in three datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The writing is good and easy to follow.\nCan you give a table to show the dimension of all the notations?\nLine 121 I do not fully understand how the computational burden reduces from NJC to (N+J)C, if h is 1~N and j is 1~J, why not  NJC for q=h+i?\nwhy do you think the view direction is useful here? how about we concatenate the rotation matrix? focal? center? or 4*4 projection matrix? why they are not useful here?\nFor body mesh recovery, what loss do you use for \\theta and \\beta regression?\nline 202 Hungarian matching is an optimization algorithm, which should not be real-time? what is your training time?\nThe group matching is a little confusing, how do you select 'consecutive' joints in line 194?\nhow is the method handle occlusion setting?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "please refer main review session", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "SxQ43eU7kc", "reviewer": "Reviewer_RDrb", "summary": "The paper presents an attentional/transformer architecture for the task of multi-view multiperson 3D pose estimation. The proposed method incorporates well-thought domain-specific adaptations of the transformer architecture for the problem at hand. It achieves very good results on standard benchmark datasets.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a transformer-based architecture for the task of multi-view multi-person 3-D pose estimation.\n\nTransformers are a good choice for 3-D pose estimation, since they overcome the limitation of high dimensionality and high memory requirement of the 3-D histogram/voxel representation.\n\nTransformers are also a good choice for multi-person pose estimation, offering an elegant way to solve the keypoint-instance association problem.\n\nThe paper proposes an architecture that capitalizes on the benefits of transformers across both axes.\n\nThe proposed method incorporates domain-specific knowledge in two smart ways:\n* Projective attention which takes into account the projective geometry of the problem and learns offsets that selectively attend to informative image areas.\n* A positional encoding via the camera ray direction which takes into account the focal length/ projective geometry.\n\nThe paper does a good experimental evaluation and ablation study of the proposed method on the CMU Panoptic and Shelf and Campus datasets. It improves compared to the VoxelPose strong baseline:\n* In terms of AP_threshold, it is significantly better than VoxelPose on the very high accuracy regime (threshold=25 mm) and very similar to VoxelPose for threshold >= 50 mm. I suspect that VoxelPose hits some quantization error limit due to its underlying Voxel representation. It is not clear to me whether this improvement is important in practice but it is good that it is there.\n* In terms of runtime, the proposed method is single shot, unlike VoxelPose whose runtime increases with the number of people in the scene. This is an important advantage of the proposed method.\n\nThe paper is accompanied by code in the supplementary material, which is good.\n\nSome minor questions that would improve the understanding of the paper:\n* Are the weights of the resnet-50 based CNN feature extractor fine tuned during training or are they kept fixed to those learned in the standard 2-D pose estimation task? How about the batch norm layers of the CNN?\n* How long does it take to train the model with the specified training protocol and hardware setup?\n\nOverall, I find this to be a well executed paper worth publishing at NeurIPS.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not very much applicable here IMO.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6Cqag6Xlmjq", "reviewer": "Reviewer_KyGs", "summary": "This work proposes Multi-View Pose transformer (MVP), a novel transformer based technique for multi-person 3D pose estimation via direct regression. There are multiple technical innovations at the core of MVP (1) a nontrivial means to embed joints as learned query features appropriate for a transformer (2) a computationally efficient projective attention technique (3) multi-view pose-specific positional encoding based on the 3D camera pose. MVP generally outperforms prior 3D multi-person pose estimation techniques. Detailed ablation studies are included.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n* High overall technical novelty - this work is the first of its kind to apply transformers to multi-person 3D. This work goes beyond a naive implementation and presents \n   * A well motivated technique to embed joints as learned query features in a transformer architecture\n   * A multi-view projective attention for aggregating information from multiple images\n   * RayConv - a camera pose-based positional encoding which significantly affects performance\n* High quality and detailed ablation studies in Section 4.3 empirically justify the authors’ design decisions.\n* Strong empirical performance compared to prior art on the Panoptic dataset.\n* High computational efficiency at inference time compared with prior work\n\nWeaknesses\n* Writing\n   * The presentation of the overall architecture can be improved. Possible suggestions are to give an overview of the method in the beginning of Section 3 that the reader can follow, or to include notation in the figures that corresponds with what’s presented in the text.\n   * More detail about mesh recovery procedure would be a good addition to the supplement. Currently it’s explained at a high level in the main text. \n* The proposed technique requires known camera intrinsics and camera pose. This potentially limits its applicability to highly controlled settings where such information is known.\n*  Transformer-based techniques are known to be computationally intensive to train, and the current approach requires 8 GPUs for training. It would be good if the authors can also comment on the total training time of the model.\n\n**Update After Discussion Period**\n\nThank you to the authors for engaging in thoughtful discussions with the reviewers. The authors addressed my concerns and no issues came up in the overall discussion that would cause me to reduce my rating. I therefore keep my initial rating: 7 - good paper, accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "2jwqFiCjjdw", "reviewer": "Reviewer_6mm2", "summary": "This paper presents an approach for multi-view multi-person 3D human pose estimation. Different from the existing state-of-the-art method [35], it does not need to localize each instance first and then estimate 3D pose for each instance. Instead, similar to the recent transformer-based object detection architecture DETR (DETR: End-to-End Object Detection with Transformers), it uses a number of N pre-learned queries (which will later be adapted based on current images) to decode N 3D poses. The main advantage of the method is that it is faster especially when the number of people is large. The accuracy is better or comparable to [35].", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper is mostly well written. One minor suggestion is to add an overview of the approach so that people can understand it more easily. In Line139-144, you used 3D joint position y when computing attention to fuse multi-view features. But according to my understanding, y is the estimation target which is not available until the last decoding layer. It will be helpful to explain how to get y in the paper.\n\n2. Section 3.4 is very similar to DETR, in particular, the bipartite matching based loss computation. The authors should mention this work and discuss the differences.\n\n3. My biggest concern lies in the generalization of the approach to different camera views and datasets. Is it possible to train on a set of cameras and test on others? What happens if we test it on completely different environment with different background?\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "-8gRH5pZiy0", "reviewer": "Reviewer_n3Zc", "summary": "This paper presents a transformer-based approach for multi-person 3D pose estimation in the multi-view setting. To achieve better performance, some novel techniques are proposed, i.e. hierarchical query embeddings, input-dependent query adaptation, projection attention, and RayConv. Hierarchical query embeddings are used to share knowledge between different persons. Projective attention is used to effectively aggregate features, and a novel RayConv is used to encode camera information in the feature maps. Experiments show the proposed approach achieves the state-of-the-art performance on Panoptic, Shelf, and Campus datasets.  \n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- This work is the first to adopt the transformer for this particular task (3d Multiview multiperson pose estimation). Although the use of the transformer is quite straightforward, this paper also introduces some novel techniques to improve upon the naive transformer baseline, i.e. hierarchical query embeddings, input-dependent query adaptation, projection attention, and RayConv.\n- This work presents a straightforward pipeline without intermediate tasks, which is not well explored recently in the literature. In the meantime, it achieves better/comparable results over two-stage approaches on main-stream datasets. \n- The paper is well written, and the main idea behind the approach is clearly explained and motivated.\n\nCons:\n•       In Table 1, the results of VoxelPose are clearly lower than those reported in the original paper [35]. In [35], AP_{25}=83.59, AP_{50}=98.33, AP_{100}=99.76, AP_{150}=99.91, and MPJPE=17.68mm. And we see that AP_{50}, AP_{100}, AP_{150} are clearly better than MvP (AP_{50}=96.6, AP_{100}=97.5, AP_{150}=97.7). This implies that the person detection accuracy is lower, and some people may be missing. The reviewer believes that the results of the original paper [35] should also be included in Table 1. The reason why the reported results are lower should be given. \n\n•       Some related works are missing:\n1) About multi-view multi-person 3d pose estimation.\n[Ref1] Zhang et al, 4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras. CVPR 2020. (Achieves 97.6 on Shelf) This paper should also be compared in experiments.\n2)  The transformers have been widely used in the task of pose estimation.\n[Ref2] Yang et al, TransPose: Towards Explainable Human Pose Estimation by Transformer\n[Ref3] Lin et al, End-to-End Human Pose and Mesh Reconstruction with Transformers\n[Ref4] Li et al, Pose Recognition with Cascade Transformers\n3) The deformable points in the transformer \n[Ref5] Zhu et al, Deformable DETR: Deformable Transformers for End-to-End Object Detection.\n•       The proposed MvP does not achieve the best performance on Shelf ([Ref1] 97.6 vs 97.4), and Campus (VoxelPose 96.7 vs 96.6). \n•       In Table 3.e, when the number of Cam is 1, the MPJPE is 93.8 which is significantly worse than VoxelPose. \n\nQuestions and suggestions:\n•       Projective Attention (Sec3.2) requires an initial 3D joint position `y` as the input. I understand that the position `y` comes from the output of the previous layer. But it is not clear how to obtain `y` in the first decoder layer.\n•       More detailed analysis (qualitatively or quantitatively) on the proposed RayConv would be nice, e.g. why RayConv can accurately capture positional information. \n•       Again on the RayConv, only camera ray directions are added to the feature map. What about the locations of the camera centers? Camera ray directions together with the camera center locations may provide more precise geometric information.\n•        More comparisons regarding runtime efficiency would be great. The only comparison is made against VoxelPose, which is computation-intensive due to 3D convolutions. However, there are many approaches in the literature and some are fast, e.g. [6]. Having such comparisons (possibly in Table 2) would better put this paper in perspective. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues found.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agreed this is an interesting paper and assigned scores ranging from 5 to 8. The rebuttal successfully clarified some of the key reviewers’ initial concerns and the ACs reached consensus that this paper can be accepted for publication. Authors are highly encouraged to address the key comments reported by reviewers in the final camera-ready version
Decision: Accept (Poster)