ContactField: Implicit Field Representation for Multi-Person Interaction Geometry
Authors: Hansol Lee
Tackgeun You
Hansoo Park
Woohyeon Shim
Sanghyeon Kim
Hwasup Lim
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D recontruction
multi-view
3D human estimation
Primary area: deep_learning_architectures
TL;DR: We introduce a novel implicit representation for interaction geometry of multiple people with the capability of simultaneously reconstructing occupancy, distinct identification (ID) tags and contact fields within a 3D space.

Abstract: We introduce a novel implicit field representation tailored for multi-person interaction geometry in 3D spaces, capable of simultaneously reconstructing occupancy, instance identification (ID) tags, and contact fields. Volumetric representation of interacting human bodies presents significant  challenges, including inaccurately captured geometries, varying degrees of occlusion, and data scarcity. Existing multi-view methods, which either reconstruct each subject in isolation or merge nearby 3D surfaces into a single unified mesh, often fail to capture the intricate geometry between interacting bodies and exploit on datasets with many views and a small group of people for training. Our approach utilizes an implicit representation for interaction geometry contextualized by a multi-view local-global feature module. This module adeptly aggregates both local and global information from individual views and interacting groups, enabling precise modeling of close physical interactions through dense point retrieval in small areas, supported by the implicit fields. Furthermore, we develop a synthetic dataset encompassing diverse multi-person interaction scenarios to enhance the robustness of our geometry estimation. The experimental results demonstrate the superiority of our method to accurately reconstruct human geometries and ID tags within three-dimensional spaces, outperforming conventional multi-view techniques. Notably, our method facilitates unsupervised estimation of contact points without the need for specific training data on contact supervision.

ContactField: Implicit Field Representation for Multi-Person Interaction Geometry
Hansol Lee hansollee@kist. re. kr
Tackgeun You tackgeun, fackgeun.you@kist.oo you@kist.
Hansoo Park hansupark@kist.re re. .r
re. re.kr kr
Woohyeon Shim 202113620sungshin. ac kc.kr
Sanghyeon Kim sangkin98@hanwha.. com
Hwasup Lim hslim@kist re kr
Center for Artificial Intelligence Research Korea Institute of Science and Technology Seoul, South Korea
Abstract
We introduce a novel implicit field representation tailored for multi-person interac- tion geometry in 3D spaces, capable of simultaneously reconstructing occupancy, instance identification (ID) tags, and contact fields. Volumetric representation of interacting human bodies presents significant challenges, including inaccurately captured geometries, varying degrees of occlusion, and data scarcity. Existing multi-view methods, which either reconstruct each subject in isolation merge nearby 3D surfaces into a single unified mesh. often fail to capture the intricate geometry between interacting bodies and exploit on datasets with many views and small group of people for training. Our approach utilizes an implicit rep- resentation for interaction geometry contextualized by multi-view loobllglobal feature module. This module adeptly aggregates both local and global information from individual views and interacting groups, enabling precise modeling of close physical interactions through dense point retrieval in small areas, supported by the implicit fields. Furthermore, we develop a synthetic dataset encompassing diverse interaction scenarios to enhance the robustness of geometry timation. The experimental results demonstrate the superiority of our method to accurately reconstruct human geometries and ID tags within dicenddiennional spaces, outperforming conventional multi-view techniques. Notably, our method facilitates unsupervised estimation of contact points without the need for specific training data on contact supervision.
Introduction
Accurate 3D representations of multi-person interactions have critical applications in virtual reality, augmented reality, robotics, and surveillance, as human subjects are central to a variety of content and tasks. In particular, modeling interactions involving multiple individuals in close proximity has gathered attention as the modeling of individual humans and simple group activities has matured. However, the precise estimation and reconstruction of 3D human body poses and shapes in close interaction scenarios present significant challenges, mainly due to occlusion, which complicates accurate reconstruction.
The Skinned Multi-Person Linear (SMPL) model (2). one of the most well-known explicit models, has been extensively utilized not only for individual human models (37 20 but also in multi-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

40]. However, as an unclothed human model, it struggles to depict local details such clothing and hairstyles. Additionally, methods using the SMPL model need separate parameter optimization for each person in a scene, which requires intricate coordination
person scenarios 8 4 39
when modeling close interactions between individuals. Implicit representations [28] 21 22] have been researched as alternatives to the SMPL. model. Implicit models, with their higher degrees of freedom, are better suited for clearly expressing local details. Successful modeling of various scenes involving individual human avatars and their detailed local features in multi-person scenarios has been achieved in [43 3 23 Nevertheless, the high degrees of freedom inherent to implicit models demand sophisticated neural architecture designs capable of handling multi-view image features and require high-quality data for training.
To address these challenges, we introduce a novel approach that represents multi pertiippesoonnnoo interaction geometries by simultaneously dealing with geometry, identity, and contact fields in scenes without the need for a prior explicit model such as SMPL. Our proposed implicit field is optimized to estimate both the occupancy and identification (ID) fields, distinguishing each person 3D space and modeling the interaction geometry. This approach enables the consideration of complex interactions between individuals while preserving the spatial information of each individual.
Furthermore, we utilize a multi-view feature transformer (5) and global scene feature extraction transformer 34 35| 36) to construct a 3D scene representation, addressing one of the biggest challenges in reconstructing close interactions: dealing with occlusions. By taking into account the global scene features for each point and leveraging latent 3D scene representations and Transformer architecture, we enhance the ability to infer information about occluded parts, which cannot be achieved with standard multi-view images alone. The Transformer, utilizing context provided by positional encoding and 3D scene representations, can infer the structure and position of occluded parts by learning from visible parts and their spatial relations. This capability is crucial for accurately reconstructing models of each person in the scene, even when direct visual information is lacking,
Additionally, we develop a synthetic dataset for multi-person interaction that includes interactions among 2, 3, and 4 individuals, which helps to address more diversity of characters and complex group dynamics in the scene.
Our experiments validate the superiority of our approach over existing methods, demonstrating its capability to accurately reconstruct and assign tag values in 3D space. Our contributions can be summarized as follows:
We introduce a novel implicit field representation for multiple people in close interaction scenarios that simultancously estimates multi- lelti--pesoonn geometries as occupancy fields, ID fields, and contact fields, thereby preserving their spatial relationships and capturing their interactions.
Our method employs novel multi-view local-global feature module coupled with global scene features extraction technique, leveraging latent 3D scene representations to reconstruct individual geometries and assign ID values and contacts in complex, occluded scenarios.
We demonstrate that our method can reconstruct 3D multi-person figures more effectively than existing methods. Also, we have created a synthetic dataset that models interac- tions among 2 to 4 individuals to enhance the understanding of group dynamics in close interactions.
The rest of the paper is organized as follows. We review the related works on 3D human representation in Section We explain method in Section 3 and demonstrate the effectiveness of the method on two datasets in Section We conclude the paper in Section 5
2 Related Work
Reconstructing 3D human models from RGB images or creating human avatars has been a longstand-
ing challenge. An explicit model, the Skinned Multi-Person Linear model (SMPL) [18] dominates
the human avatar research by serving as a canonical 3D human model [37 15 14
30
20 6].
However, SMPL is an unclothed human model and is limited in its ability to capture local details such as clothing and hairstyles. Hence, implicit representations, including signed distance fields (SDF) 28 and occupancy fields [4). have gathered attention from the community. Pixel-aligned

---

Input polor X
Local Foatars
Maltistew Posture Paaken Modele
Image Ensnder f
Geornetyy
FP F
Image
Occepency Fied
Glubal Feature
Normal Maps (
Maltivsio Iroge Y
Spene Encoder
2
Decoder
MELP
Contact Fidd
Ray Directions (4
Schatce Socre Represestrzinn
Globel Feature
Canera Positions
(o
Implicit Fields Interaction Goounttry
9
ID Stald
Figure 1: The overall framework of our method. We compute the local and global features from set of multi-view images and its camera parameters through the proposed multi-view feature local-global transformer. We use local feature to estimate occupancy and global feature to estimate ID at given point x. From the occupancy and ID fields, we estimate the contact field, as detailed in Section 3.4
contextualize the implicit fields by projecting image features onto 3D coordinates using the camera parameters to enhance 3D human avatar reconstruction. Utilizing multi-view RGB images effectively mitigates occluded parts of human in single-view RGB images. DoubleField (38) integrates neural surface and multi-view based radiance fields to represent 3D geometry and the appearance of humans.
feature encoders
Multi-person 3D reconstruction presents unique challenges not encountered in single-person scenarios, such as occlusion, where subjects may obstruct each other's visibility. This task requires not only detailed understanding of spatial relationships among individuals but also the preservation of individuality while representing their interactions. This process needs precise reconstruction and a deep interpretation of interactions. MPSD 23 employs an implicit approach for the 3D reconstruction of each person, utilizing 6-DOF spatial position estimation within the global scene space. Their method enables muulti-prrson 3D reconstruction from single image, effectively tracking the locations of individuals. However, because this method primarily addresses data where individuals are spaced widely apart, it does not adequately handle scenarios involving people close interaction.
42 3 are a crucial challenge in multi-person representation. Deep- MultiCap [43) incorporates an attention module and temporal fusion produce high-fidelity 3D models, relying on the SMPL model for prior segmentation, prediction, and location determination. However, this approach faces challenges, as the accuracy of 3D reconstruction and spatial estimation is bounded by the performance of an initial SMPL prediction, and it requires multiple optimization steps for each individual instance. The work (3] leverages single-view image and gDNA (7) generative model to generate 3D geometry. refining the spatial positioning of each individual based on contact information. Considering that both methods sequentially reconstruct 3D geometry for each individual, accurately representing multiple subjects in close interaction remains challenge. Also, existing methods for 3D reconstruction of close human interactions often suffer from data scarcity. The introduction of Hi4D [42] marks leap forward, offering detailed 4D textures and essential data for studying two-person interactions. However, Hi4D (42) includes only two people, while the dataset described by MultiHuman 43 features up to three. In these two datasets, individuals are captured either separately or together, often overlapping or passing through each other.
Close interaction scenarios 43
Our method innovatively addresses these complexities by employing transformer-basee architecture that integrates multi-vien feature fusion with global scene representation, allowing the simultaneous and dynamic reconstruction of multiple interacting individuals. This approach not only captures the detailed geometries of each person but also maintains their unique identities and spatial relationships, even in scenarios with significant occlusions and close physical interactions. Additionally, we created a synthetic dataset that models interactions among 2, 3, and individuals to address even more complex group dynamics.
3 Method
Our method introduces a novel representation of multi-person interaction geometry by combining 3D reconstruction with identification of multiple individuals in close interaction from multi-view
3

---

images. We achieve this by estimating two key fields: occupancy and ID. The occupancy estimation crucial for reconstructing the geometry of each person, while the ID estimation facilitates the identification of individuals within the 3D space. We leverage transformer architectures to tackle the significant challenge of occlusions common in close interaction scenarios. Our architecture first extracts local features and then computes global features by integrating these local features with 3D scene representations. This approach ensures robust integration of local and global information across multiple perspectives, enabling comprchensive understanding and reconstruction of the complex configurations of scenes. Figure S represents our process pipeline, encapsulating the essence of our approach.
3.1 Implicit Fields for Multi-Person Interaction Geometry
Our method utilizes implicit functions to represent interaction geometry in 3D space. Implicit functions uniquely define the surface of 3D object by specifying level set within a field (2T). This representation allows for a continuous definition of the surface, enabling the precise reconstruction of complex geometries. Alongside the geometric reconstruction, we introduce a novel approach to model the ID) and contact fields within the same 3D space, providing a method for distinguishing individual entities closely interacting scenarios.
Our model takes query point x E R to predict two key attributes: the occupancy s, and the identification q for each query point. We extract multi-view local and global features, denoted as (-)
obk $2], given set of multi-view images { their corresponding image normal maps [N") J==1 and camera parameters {K") where V denotes the total number of views. The proposing model g(-) can be formally defined by the following equation:
8, g(x; (x((II,N" K"",""' K")W))
(I)
The occupancy value E 1] is binary indicator signifying whether point resides inside or outside the surface boundary of an individual, essentially distinguishing the geometric presence of the subject in the 3D space. The identification (ID) value q E RR provides unique identifier each point, allowing differentiation of individuals in close proximity by assigning distinct ID values.
3.2 Multi- Local-Global Feature Module
The architecture of the 4x module, which extracts local and global features [ Φ2 from multi- view images, determines the quality of occupancy and ID fields. We introduce a local-global feature scheme through a dedicated feature extraction architecture.
Given a set of images I", normal maps N generated by the method described in IntegratedPIFu (5) and camera parameters K", our method begins by following PIFu (32) using an image encoder [25] Each image and normal map extracted by [5] is processed individually within the multi-view inputs, allowing distinctive features be captured from different perspectives. For any given query point in 3D space, we project this point onto the 2D planes of the multi-view inputs to acquire pixel-aligned features. Formally, the feature extraction process can be described as follows:
F f + N*), F [[F", K", x), Ve E (1.2...,
where F" denotes the set of features extracted from concatenated w-th image and e-th normal map and ( symbolizes channel-wise concatenate operation. fime represents the image encoder function, designed to process each views.
F" represents the features at the image pixel corresponding to the projection of point x onto the e-th image plane. The function II(-) computes the location on the image plane where the 3D point is projected and extracts the relevant features from F at that point. For the detailed implementation, refer PIFu 322.
This approach ensures that the features F are aligned with the geometry of the scene as observed from multiple viewpoints, facilitating an accurate reconstruction of the 3D space.
The pixel-aligned features extracted from each view are then aggregated through a local-global process to create comprehensive feature representation for each query point This aggregation is

---

performed using a view-to-view transformer encoder (38). formulated as:
D (x(x), 1 F2 ,)),
(4)
where D% represents the local feature set obtained by fusing features across all views corresponding to the query point x with positional encoding 7x).
Inspired by the architectures of the Scene Representation Transformer (SRT) encoder [34 and decoder mechanism [36], the global feature module complements these local features with global scene context. The SRT encoder f receives the extracted features from all views along with their corresponding camera positions o and normalized ray direction d from each camera information to encapsulate global scene information into compact representation z. This scene representation serves as an input to the global feature decoder fod, which extracts global features 46;
== f ({F o", d=),,, = zz 24),
(5)
(6)
where & represents the global features for multi- images.
Finally, we forward the features (x(+) into two multi-layer perceptrons (MIPs), oe and de to retrieve occupancy values and ID values for each query point:
(
(7)
(
(8)
This method ensures a comprehensive feature set that enhances predictions by integrating both localized and globalized scene insights.
Detailed architectures of fe, fse, and fad shown in section A.2 of the Appendix.
3.3 Training Objective
We train this model y with following objectives:
C w. LMSE + Weontrafcomrra Wgroup Carupp
where CMSE is used to train occupancy field, while Lcontra and Lgroup focus on accurately identifying individual entities captured ID fields.
For occupancy predictions, mean Squared Error (MSE) loss is defined for N query points as:
N
1
MMSE
Σ [s - )
(10)
N
where is the predicted occupancy and s is the ground truth for the i-th point. LuSE reconstructs 3D geometries by minimizing the discrepancy between the predicted and actual occupancy values, crucial for capturing the intricate details of the scene.
To ensure that points associated with the same object are assigned identical predicted ID values, we tailor an associative embedding (24) consisting of contrastive loss [9] 12 and grouping loss in the training objective. The contrastive loss is computed based on pairwise Euclidean distances among ID value, considering both positive pairs for the same instance label and negative pairs for different instance label. For all query points {x:} J. we have a set of predicted ID values {q: and associated ground truth instance labels {l(x;)} given from datasets. Then, the contrastive loss is formulated as:
Eng(m pas dey) + Eis (m
neg max(0, -di + +))
Leontra
(11)
P
where di = 1|g: - Oj is a pairwise Fuclidean distances between ID values of and j-th query points. m = I[4 = 1,] and neg = ITZ,, + 1y] are mask value for indicating positive or negative pairs. is a predefined margin threshold. and that returns 1 for true and returns 0 for false case. If 5 = 1, then the negative loss component is doubled. This function aims to minimize the distance between positive pairs while ensuring negative pairs separated by at least the margin.
the total number of pairs. [[[ denotes an indicator function
5

---

The grouping loss function is devised to refine the embeddings of predicted ID values by accomplish- ing two primary objectives: minimizing the variance within groups of values that correspond the same ground truth label and maximizing the separation between groups linked to different labels. This function articulated as follows:
K
K + 1= t= 1iik
Ggroup =
Σ [xx)GG
(x
m
(12)
G
k=
In this equation, K denotes the total number of unique ID) values present in the ground truth labels, signifying the distinct classifications for the predicted ID values. Each G3 represents the collection of predicted ID values that match the k-th unique ground truth label, forming groups of predictions intended to bear the same ID. The term |G& reflects the size of each group. indicating the count of predictions it encompasses. The variable x refers to individually predicted ID value within the group GR. and μ4; is the average of values in Gk. acting as the group's centroid. The notation | signifies the absolute value, ensuring that distances the computations remain non-negative.
The first component of the grouping loss concentrates on reducing the squared distances between each predicted ID value and its group's 's mean M4x. thereby decreasing the variance within each group. Conversely, the second component introduces an exponential penalty on the closeness of centroids 14 and l4t from distinct groups, fostering clear separation between these groups by penalizing closely situated centroids. This bifocal approach of the loss function is crucial for steering the model towards generating cohesive yet distinctly separated embeddings, which fundamental for the precise identification and differentiation of unique ID values.
3.4 Estimation of Interaction Geometry
To reconstruct the geometry of occupancy fields, first retrieve the occupancy fields by querying dense points from the implicit fields 9- After estimating the occupancy field, the geometry is reconstructed using the Marching Cubes algorithm [19]. Initially, define the bounding box of the voxel grid. The Marching Cubes algorithm is then applied to the grid, where a surface is generated by interpolating the predieted occupancy values with the algorithm retrieves the ID values from the same voxel units and applies a color map, which facilitates the visual distinction of different individuals represented in the geometry. We 7 0.5 for thresholding occupancy values.
within each voxel unit. Subsequently,
The final contact fields be estimated by the variance of predicted ID values. Close interaction among different instances typically leads to deficient information due to occluded images. Conse- quently, the uncertainty in the predicted geometry is increased when the information from multiple views is deficient. We choose the variance of predicted ID values among possible options as an uncertainty metric because Euclidean distance is adopted within the training objective for predicted ID values. For each voxel x = (x), 22, 2z) in the 3D grid V, we extract a local neighborhood, N ((x), excluding background value (s T). We then calculate the standard deviation of the neighborhood values,
0x
[(x)
(v 1vv-14 14)2 vEN(x)
Σ
(13)
where μ is the mean of the neighborhood values and [V(x) is the number of elements in the neighborhood.
A voxel is marked as contact point if the standard deviation exceeds a threshold of Te = 0.25. The formulation as follows:
c(x) = ={0-
[xx if 0xx >
(14)
otherwise
3.5 SynMPI: Synthetic Dataset for Multi- Interaction
Current multi-human benchmark datasets, such as Hi4D 42 and MultiHuman 43 are limited in size and scope, particularly in terms of the number of interacting individuals and the diversity of interaction scenarios. To address these limitations, introduce synthetic dataset designed

---

congle doncing
gransioutter vuiching days
apgrrmach
sosr rus GIIO?
a couple lving and salting (a) Two-person
ds aigry Mos wilh tru Homes on their phoner
mun plasing gooar iurrusnie by dheee poople Four-person
(b) Three-person
Figure 2: Examples of multi-person interaction geometry in the SyлMPI dataset. Each sample contains interactions involving (a) 2, (b) (c) people. In each sample, the left side shows rendered RGB images and the right side shows rendered meshes. Italic sentences explain the multi- people interaction types of samples adopted from Character Creator 4 31
to encompass a broader range of interaction scenarios involving groups of up to four individuals. Figure shows examples from our dataset.
Our dataset is multi-viev and supports multi-person interactions, accommodating groups of two to four individuals to effectively capture the spatial dynamics of group interactions. We include elderly individuals and children, representing wider range of identities for the enhanced diversity of dataset. Additionally, the synthetic data features individuals with varying ages, heights, weights, and clothing styles. We also incorporate multiple types of motions for each participant, enriching the dataset with diverse set of dynamic interactions. The dataset encompasses approximately 50 distinct motion types, including dancing, running, and talking. The dataset includes 49 characters (26 female, 23 male) with 50 motion sequences across 43 scene configurations, totaling approximately 25,000 frames. For detailed breakdown of the dataset, please refer to Appendix(B... B.I
4 Experiments
4.1
Datasets
We conduct our experiments on Hi4D [42 and the proposed SynMPI dataset, which include various multi-person interactive scenarios. Aggregated training splits of both Hi4D and SynMPI serve as the training set for our experiments. The test splits of each dataset serve as the test set.
Hi4D (42 This dataset offers samples which individuals engage in various interactions, encom- passing 20 subject pairs with diverse body shapes and appearances with 8 view images. It includes 3D human scans, instance segmentation masks at the vertices of the 3D scans and image pixels, SMPL (2] parameters, and contact information at the vertex level. We use a random split of 70% for training and 30% for evaluation.
SynMPI Our synthetic multi-person interaction dataset captures a wide range of interaction scenar- ios involving groups of more than two individuals, encompassing diverse spectrum of dynamic interactions. From the our synthetic datasets, we use images from views and 3D geometry for our experiments. We randomly split samples in SynMPI into 70% for training and 30% for evaluation.
4.2 Metrics
we adopt four metrics assess quality of the interaction geometry. Chamfer Distance (CD) calculates the bidirectional disparity between points on the predicted and corresponding ground-truth mesh. Point Surface (P2S) computes the unidirectional distance from each point of the ground-truth mesh to the nearest surface
Following the evaluation protocols of existing studies 32
43

---

Table I: Evaluation results for multi-person interaction geometry. The values presented under the CP row indicate the threshold value, denoted by c, which was employed to construct the pseudo G.T. contact map in 3D space for evaluation.
Hi4D
42
SynMPI (Ours)
Model
CP
CD1
P2S.
NC1
CD1
P2SJ
NC1
0.025
0.05
0.075
0.1
DMC Ours (w7o SRT)
0.631
0.495
0.768
0.804
0.800
0.688
0.4688
"0.402
0.888
0.317
"0.424"
00.438
0.492
0.630
0..92
0.827
Ours
0.406
0.329
0.892
0.447
0.629
0.670
0.703
0.511
0.374
0.836
Table 2: Ablation study on grouping loss function in Eq
CP1
Ablation type
Squared distance
Exponential penalty
CD4
P2SL
NCT
0.05
0.075
used (b) used
used
0.462
0.363
0.892
0.111
0.187
not
0,400
0.314
0.892
0.345
0.528
(c)
not used
used
0.532
0.403
0.880
0.228
0.335
(d)
used
used
0.406
0.329
0.892
0.629
0.670
on the corresponding predicted mesh based on the closest-set euclidean distances. Normal Consistency (NC) measures the difference the normal vector between points on the predicted and corresponding ground-truth mesh with the nearest-set euclidean distances. Contact Precision (CP) is defined by the overlap between the estimated contact map and the pseudo ground truth contact map generated from ground truth meshes, For detailed definition of the metrics, please refer to the Appendix A.4
4.3 Baseline Models
43 reconstructing 3D human with multi-view images. DMC leverages 3D feature of SMPL mesh to infer information of occluded regions during the learning process of the pixel-aligned implicit function. For the waasformer-basee approach. They
DeepMultiCap (DMC) [33] We compare our framework with existing methods
integration of features from multiple views, it utilizes 8-view images for their reconstruction process, and we followed the same setup. We use the public implementation of DMC apply LVD (10) the SynMPI, well MVpose [11] on the Hi4D, to obtain SMPL parameters. We refer to Appendix B.2 for additional details.
4.4 Results and Analysis
Quantitative Results Reconstruction results are evaluated against the baseline method, as shown in Table Our method demonstrated superior performance in terms of reconstruction quality metrics, indicating the effectiveness of our approach in accurately capturing and reconstructing 3D models of multiple people.
Qualitative Results Figure illlsttrates the qualitative performance of our method in generating high- quality reconstructions of multiple people in close interaction scenarios. Compared to DMC (43), our method excels in handling dynamic interactions and heavy occlusion, which are common challenges in multi-person reconstruction tasks. DMC struggles with these scenarios, leading to less accurate SMPL estimations. For additional results, please refer to the appendix and supplementary video.
Ablation Study on Architecture We performed ablation studies to assess the impact of our proposed modules. Table I presents the performance of the geometry module without the global features. The results demonstrate that global features significantly enhance performance across all geometry- related metrics. Table [ highlights the impact of global features contact precision performance, demonstrating enhanced accuracy of contact predictions across range of thresholds. visurlyy visually compares contact precision performance with and without global features, illustrating substantial

---

Table 3: Ablation on the number of views.
Table 4: Method for contact map estimation
Model
views
CD4
P2S.
NC1
CPf
Method
DMC 43
4
1.304
0.922
0.705
0.05
0.075
8
0.631
0.495
0.768
4
0.761
0,472
0.870
output meshes
0.621 0.629
0.518
Ours
8
0.406
0.329
0.892
variance estimation (ours)
0.670
the geometry performance with ID and ID field volume rendering, further demonstrating the positive impact of global features.
improvements.
Additionally,
Ablation Study on Grouping Loss Grouping loss function defined in Eq 12 comprises two key terms: the first is the squared distance, and the second is the exponential penalty. In Table 2 model (a) is trained without the grouping loss, while model (d) is trained with the grouping loss. The exponential function in the second term encourages soft assignment to specific instance cluster. However, using only this term does not lead to improved grouping performance. We observe that the combination of both terms within the grouping loss function results in overall performance enhancement.
Contact Map Analysis Table also presents the performance of our contact map using contact precision metrics. Our approach leverages unsupervised learning to predict contact fields in 3D space, eliminating the need for labeled training data, which is often difficult and costly to obtain. This particularly advantageous in complex scenarios involving multiple individuals and interactions. The effectiveness of our method is significantly influenced by the resolution of the 3D data. High- resolution data provide detailed and dense information, enabling precise contact prediction. In contrast, low-resolution data can lead to less accurate results due to the sparse representation of the interactions, as illustrated Figure A key advantage of our approach is the use of an implicit contact field, which allows for flexible changes in resolution. This flexibility enables method to adapt to various data resolutions without compromising the integrity of the contact prediction. Thus, method excels in flexibility and reduces dependency on extensively labeled datasets. However, ensuring adequate resolution of 3D data is crucial for achieving optimal accuracy and reliability in contact field estimation. this paper, we use resolution of 256 estimate the contact field.
To further assess our contact predictions, we directly infer contacts from the output instance meshes by examining geometric proximity and the surface identifiers used generate the contact map from the pseudo ground truth instance meshes. Table 4 presents the results for both contact map estimation methods. Our variance-based estimation of the contact field in 3D space yields better results than the mesh-based inference. This is because the variance estimation method benefits from high-resolution 3D data and operates in continuous space, allowing for precise localization of contact areas without intermediary steps that could introduce errors. In contrast, the mesh-based inference relies reconstructed meshes that may lack fine details due to resolution limitations or reconstruction errors, leading to less accurate contact predictions. Additional details are provided in Appendix B.3 Ablation Study on the Number of Views Table presents an ablation study on the number of views. Our method consistently outperforms DMC across all metrics Chamfer Distance (CD), Soint-to-Surface Distance (P2S), and Normal Consistency (NC) in both the 4-view and 8-view settings.
metrics-CCamfer
Conclusion
This paper addresses the intricate challenges associated with the 3D reconstruction of multiple interacting human bodies in close proximity, an area critical for applications in virtual reality, augmented reality, robotics, and surveillance. Our approach overcomes the limitations of traditional methods that rely on models like the Skinned Multi-Person Linear (SMPL), which often struggle in scenarios with dynamic interactions and occlusions. By employing advanced methodologies, including multi-view feature transformer and a global scene feature extraction transformer, our approach not only preserves the unique identities and spatial information of each individual but also enhances the accuracy of 3D reconstructions.

---

Impot Tmages (Front view)
(Geometry)
(Geumetry)
(ID)
(Geometry)
Back view
Figure 3: We compare our method baseline DMC (33 on Hi4D (top) and SynMPI (bottom) test set. From left right columns, we show the input mieti-view images, the generated geometry by each method, and ground truth (GT) Geometry.
Interaction Geometry
Volume Rendering
G.T Instance Meshes
Ours SKT)
Own
Figure 4: Visualization of reconstructed multi-person interaction geometry and instance-wise volume rendering with ID fields for visualizing occluded regions during interaction.
Input Images
Inpul Images
resolution 128")
Figure 5: Comparison of the effect of global features and 3D resolution on estimated contact fields.
Ours (w/o global) excludes global features, Ours (low resolution 1283) uses low resolution.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by Korea Creative Content Agency (KOCCA) grant funded by the Korea government (MCST) (No. RS-2023-000224227, Development of intelligent virtual sports service technology for breaking dance)..
References
[1] Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu. "Photorealistic monocular 3d reconstruction of humans wearing clothing". In: CVPR. 2022.
[2] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. "Keep SMPL. Automatic estimation of 3D human pose and shape from single image". In: ECCV. 2016.
[3] Junuk Cha, Hansol Lee. Jaewon Kim, Nhat Nguyen Bao Truong, Jaeshin Yoon, and Seungryul Back. "3D Reconstruction of Interacting Multi-Person in Clothing From a Single Image". In: WACV. 2024.
[4] Junuk Cha, Muhammad Saqlain, GeonU Kim, Mingyu Shin, and Seungryul Baek. "Multi- Person 3D Pose and Shape Estimation via Inverse Kinematics and Refinement". In: ECCV. 2022.
[5] Kennard Yanting Chan, Guosheng Lin, Haiyu Zhao, and Weisi Lin. "Integratedpiftu: Integrated pixel aligned implicit function for single-view human reconstruction". In: ECCV. 2022.
[6] Dongyue Chen, Yuanyuan Song, Fangzheng Liang, Teng Ma, Xiaoming Zhu, and Tong Jia.
"3D human body reconstruction based on SMPL model". In: The Visual Computer (2023).
[7] Xu Chen, Tianjian Jiang, Jie Song, Jinlong Yang, Michael Black, Andreas Geiger, and Otmar Hilliges. "gDNA: Towards Generative Detailed Neural Avatars". In: CVPR. 2022.
[8] Hongsuk Choi, Gyeongsik Moon, JoonKyu Park, and Kyoung Mu Lee. "Learning to estimate robust 3D human mesh from in-the-wild crowded scenes". In: CVPR. 2022.
[9] Sumit Chopra, Raia Hadsell, and Yann LeCun. "Learning a similarity metric discriminatively, with application to face verification". In: CVPR. 2005.
[10] Enric Corona, Gerard Pons-Moll, Guillem Alenya, and Francese Moreno-Noguer. "Learned vertex descent: A new direction for 3d human model fittiig".. In: ECCV. 2022.
[11] Junting Dong, Wen Jiang, Qixing Huang, Hujun Bao, and Xiaowei Zhou. "Fast and robust multi-person 3d pose estimation from multiple views". In: CVPR. 2019.
[12] Raia Hadsell, Sumit Chopra, and Yann LeCun. "Dimensionality Reduction by Learning an Invariant Mapping". In: CVPR. 2006.
[13] Tong He, John Collomosse, Hailin Jin, and Stefano Soatto. "Geo-pifu: Geometry and pixel aligned implicit functions for single-view human reconstruction". In: NeurlPS. 2020.
[14] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. "End-to-End Recovery Human Shape and Pose". In: CVPR. 2018.
[15] Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis. "Convolutional Mesh Regression for Single-Image Human Shape Reconstruction". In: CVPR. 2019.
[16] Jungeun Lee, Sanghun Kim, Hansol Lee, Tserendorj Adiya, and Hwasup Lim. "PIDiffu: Pixel-Aligned Diffusion Model for High-Fidelity Clothed Human Reconstruction", In: WACV. 2024.
[17] Stuart Lloyd. "Least squares quantization in PCM" In: IEEE transactions on information theory (1982).
[18] Matthew Loper, Naureen Mahmood, Javier Romero. Gerard Pons-Moll, and Michael J Black. "SMPL: A skinned multi-person linear model".. In: TOG (2015).
[19] William E. Lorensen and Author PictureHarvey E. Cline. "Marching cubes: A high resolution
3D surface construction algorithm" " In: Computer graphies (1987).
[20] Yang Lu, Han Yu, Wei Ni, and Liang Song. "3D real-time human reconstruction with a single RGBD camera". In: Applied Intelligence (2023).
[21] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. "Occupancy networks: Learning 3d reconstruction in function space". In: CVPR. 2019.

---

[22] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis". In: Communications of the ACM (2021).
[23]
Armin Mustafa, Akin Caliskan, Lourdes Agapito, and Adrian Hilton. "Multi-person implicit reconstruction from a single image". In: CVPR. 2021.
[24] Alejandro Newell, Zhiao Huang, and Jia Deng. "Associative Embedding: End-to-End Learning for Joint Detection and Grouping". In: CVPR. 2016.
[25]
Alejandro Newell, Kaiyu Yang, and Jia Deng. "Stacked hourglass networks for human pose estimation". In: ECCV. 2016.
[26] NVIDIA. Kaolin wwtpt:tttt:...tt.
[27]
NVIDIA. Omniverse USD Composer. https // docs omniverse nvidia com composer/latesttnndee
html/
[28] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. "Deepsdf: Learning continuous signed distance functions for shape representation". In: CVPR. 2019.
[29] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. "Automatic differentiation in PyTorch". In: NeurIPS 2017 Workshop on Autodiff. 2017.
[30] Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas Daniilidis. "Learning to estimate
3D human pose and shape from single color image". In: CVPR. 2018. Reallusion. CharacterCreatorv.
[31]
pettps:/wwit http://wwwweealsiiiiooooo www.lrrlllsio.. www wtt.//ccerrrrratttoooo
[32]
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. "Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization". In: ICCV, 2019.
[33] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. "Pifuhd: Multi-level pixel- aligned implicit function for high-resolution 3d human digitization". In: CVPR. 2020.
[34] Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. "Object scene representation transformer". In: NeurIPS. 2022.
[35] Mehdi SM Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Lučić, and Klaus Greff. "Rust: Latent neural scene representations from unposed imagery". In: CVPR. 2023.
[36] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, al. "Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations". In: CVPR. 2022.
[37] Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. "Probabilistic 3D Human Shape and Pose Estimation From Multiple Unconstrained Images in the Wild". In: CVPR. 2021.
[38] Ruizhi Shao, Hongwen Zhang, He Zhang, Mingjia Chen, Yan-Pei Cao, Tao Yu, and Yebin Liu. "Doublefield: Bridging the neural surface and radiance fields for high-fieeltyy human reconstruction and rendering". In: CVPR. 2022.
[39] Yu Sun, Wu Liu, Qian Bao, Yili Fu, Tao Mei, and Michael J Black. "Putting people in their place: Monocular regression of 3d people in depth". In: CVPR. 2022.
[40] Tao Wang, Jianfeng Zhang, Yujun Cai, Shuicheng Yan, and Jiashi Feng. "Direct Multi-view Multi-person 3D Pose Estimation". In: NewrIPS. 2021.
[41] Chenyan Wu, Yandong I., Xianfeng Tang, and James Wang. "MUG: Multi-human graph network for 3D mesh reconstruction from 2D pose". In: arXiv preprint arXiv:2205./2583 (2022).
[42] Yifei Yin, Chen Guo, Manuel Kaufmann, Juan Jose Zarate, Jie Song, and Otmar Hilliges. "Hi4d: 4d instance segmentation of close human interaction". CVPR. 2023.
[43] Yang Zheng, Ruizhi Shao, Yuxiang Zhang, Tao Yu, Zerong Zheng, Qionghai Dai, and Yebin Liu. "Deepmulticap: Performance capture of multiple characters using sparse multiview cameras". In: ICCV. 2021.
[44] Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu. "DeepHuman: 3D Human Reconstruction From a Single Image". In: ICCV, 2019.
12

---

A Implementation Details
A.1 Training
We trained model the Hi4D 42 dataset and our dataset SynMPI. Following the PIFu (32 process to extract pixel-aligned features, we sampled 6000 points used during training. The training was conducted with batch size of 4 and learning rate of le- Our learning rate schedule involved decaying the initial learning rate by a specified factor (gamma) at predetermined epochs, as defined in our schedule. The model was optimized using the RMSprop optimizer. In our experiments, set = 1. Wecnttr = 0.1, and Wgrooup 0.1 for the weights of loss functions. Implemented using PyTorch (29). the entire training process spanned approximately two days and covered 100 epochs using two NVIDIA A100 GPUs. Inference of each instance requires around 60 seconds on the same GPU.
A.2 Architecture
Multi- View Feature Fusion Module f". Our objective with f effectively aggregate features from multiple view inputs. Utilizing a view-to-view transformer architecture inspired by Double- Field (38], process pixel-aligned features facilitate this aggregation. The operation of our transformer is mathematically represented as:
O Kt. V = E. ) $ s (Attention Q% KG V()),
(15)
where Qi, KL, and Vid denote the query, key, and value matrices generated from the input features, respectively. Following self-attention, the features are further refined through feed-forward networks to obtain the local feature set /b
f (
SRT Encoder f SRT Encoder, denoted as f aims to encapsulate 3D scene information into comprehensive set-latent scene representation 2. Following methodologies similar to those described by Object Scene Representation Transformer (OSRT) (34] our encoder leverages the Transformer's s self-attention mechanism to aggregate spatial and feature information from multiple views into single scene representation. This process is formalized as:
(F.) = EF(( (o", do"""""""""""""""""""" dd)))=
(16)
where e and Gray represents the conv block yielding set of features {Fo)y Subsequently, these features aggregated into set of flatted patch embeddings {E, where N denotes the total number of patches across all images. This aggregation can be mathematically represented as:
{
Sanch ((Fo)) JY h=m));
(17)
The transformer encoder, T then processes this set of embeddings to generate the final set-latent scene representation 2:
z ({E.))))
(18)
where z fully encapsulates the observed 3D scene, encoding comprehensive spatial and visual scene information. It is imperative note that set-latent scene representation embodies the comprehensive understanding of the specific 3D scene as observed through the corresponding set of images. This representation, characterized by its ability to maintain the integrity and richness of scene's 's spatial and feature information, is pivotal for subsequent reconstruction and analysis tasks.
Global Feature Decoder fsd, We employ SRT decoder to extract global features from scene representation 2. There are some differences with original SRT decoder (36). The main difference is that local feature &h is used for query and value in multi-head attention mechanism. This modification ensures that:
C - fad TL..
(19)
where &C represents the globally decoded feature. This enables the decoder to dynamically focus on relevant scene information, thereby facilitating detailed and accurate 3D reconstructions and precise occupancy and identification predictions.
13

---

A.3 Dataset Construction
To create our dataset, we first acquired characters of various ages and interaction motion sequences from Character Creator 4 [31]. We then composed scenes featuring multiple characters using Omniverse USD Composer [27 To facilitate the dataset generation process, we modified the Kaolin rendering tool (26) to include tasks such as multi-person normalization, enabling us to achieve the desired outputs. This process allows us generate multi-view rendered images, mask images, instance masks, and 3D geometry.
A.4 Evaluation Metrics
The definition of evaluation metrics are shown in below. P and Q refer to the set of 3D points.
Chamfer Distance (CD) This metric calculates the bidirectional disparity between points on the predicted and corresponding ground-truth mesh. It computes the euclidean distance from each point to nearest surface on other mesh. Lower value of CD metric indicates higher fidelity of reconstruction.
CD Q)
1 P
min lpp DEP q
Σ
q/l2 +
Σ qEQ
min ||q - pl|
(20)
Q
PEP
Point to Surface (P2S) This metric computes the unidirectional distance from each point of ground- truth mesh to the nearest surface on the corresponding predicted mesh based on the closest-set euclidean distances. Lower value of P2S metrie means superior reconstruction accuracy.
P2S Q)
1
min |p - -ll2 pEP qEe
Σ
(21)
P
Normal Consistency (NC) This metric computes the difference of normal vector between points on the predicted and corresponding ground-truth mesh with the nearest-set euclidean distances. It uses the bidirectional way for calculating the difference. Lower value of NC metric indicates a higher fidelity of reconstruction.
NC(P, Q)
1
E -p Dreares((pp.)) +
Σ ng Dnearest(q.PP)
(22)
2 P
2/
Contact Precision (CP) We first identify sample points in the voxel grid using the ground truth mesh Mgt.
P Pinside = (p E P Mg contains p}-
(23)
For each point p; P Pinside. determine the nearest points the mesh surface M and assign surface identifiers 8; based on face indices ft. Using a KD-Tree, we find neighhoring points within a specified distance threshold 6,
N (py Il. pyll < and j * i).
(24)
contact marked if any neighboring point has a different surface identifier,
{
if Dp, E N; such that 8j +
contact,
(25)
0
otherwise
This process results in contact labels that are contextually relevant to the mesh's surface features, enabling validation of our contact prediction algorithms using precision as the evaluation metric.
The contact precision defined by the overlap between the estimated contact map E and the pseudo ground truth contact map T generated from ground truth meshes. Then, the precision is given by
T E
P(T,E)
(26)
E
where 'T E is the number of true positives (correctly predicted contact points), and |E] is the total number of predicted contact points. A higher precision value indicates that greater proportion of the contact points predicted by the model are correct, signifying fewer false positives. This metric is crucial for assessing the accuracy our contact prediction algorithms, ensuring that the predicted contacts closely align with the contacts defined by the pseudo ground truth.
14

---

Table A: Evaluation excluding SynMPI dataset on Hi4D test sets
Genmetry
Contact Precision f
CD1
P2S.
NCT 0.025
0.05
0.75
0.1
w/o synthetic data
0.499 0.418
0.885
0.351
0.482
0.514
0.542
w/ synthetic data
0.406 0.329
0.892 0.447 0.629
0.670
0.703
Table B: Ablation study on SPML initialization method for synthetic datasets in DMC
Model SMPL. method
CDL
P2SJ
NC1
DMC
MVPose [11]
0.805
0.489
0.771
DMC
LVD T0
0.631
0.495
0.768
Gender
Scene Configuration
Age Distribution
Distribution Charactars Age Scones
ca
4.7
1L25
Elselty Eisatty (Foaal)
22.4%
Aduit (ttane
of
AcUlt ci
Figure A: Statistics of the SynMPI dataset.
B Analysis
B.1
Datasets
Table A presents the results of an ablation study using our SynMPI dataset. Training models with SynMPI alongside Hi4D (42 led improved performance on the Hi4D test set. This improvement is mainly due to the large diversity within our dataset, which includes variations in age, gender, and scene composition, as shown in Figure A
B.2 Baseline Models
For the baseline models discussed in Section we employed different SMPL. (18) acquisition methods tailored each dataset. Specifically, for the HI4D dataset, we used MVPose [I], following the experimental setup described in the HI4D paper, where DMC [43 was run using MVPose for SMPL acquisition. We adopted this approach to ensure consistency and comparability.
Table B presents the ablation study on initial SMPL methods. However, we encountered challenges when using MVPose for our synthetic dataset, its modules were trained on real data, leading less accurate SMPL estimations on synthetic data. To address this, we utilized Learned Vertex Descent (O), which is designed to fit SMPL 3D human model (3D scan) and has proven to provide more accurate results in this context. Although LVD is originally intended for single-person scenarios and may be sensitive to occlusions, we selected for the synthetic dataset achieve accurate results in our study.
B.3 Contact map
We also measure contact precision using the generated instance meshes as well as our proposed variance estimation of contact fields. To provide further insights into method, explain how different individuals are distinguished using predicted ID values during mesh generation.
To generate instance meshes from our implicit fields, our algorithm identifies and marks regions of interest based on the occupancy field during inference, excluding the background. This process generates both ID field and contact field. The normalized ID values within these regions
15

---

Input Imagess
ID
Instance Moshes Instance Meshes
Input Images
Instance Moshes
HIHD Instance Meshes
Figure B: Example of Instance meshes.
Input Images
Interaction Geometry
ID Field Volume Rendering
G.T. Instauce Meshes
Ours (wle SRT)
Ours
Figure C: Visualization of reconstructed multi-person interaction geometry and instance-wise volume rendering with ID fields for visualizing occluded regions during interaction.
grouping the data into clusters, with each cluster representing a different individual and including the associated contact regions.
are processed using k-Means clustering 17
After clustering, each cluster is isolated with binary mask, which is smoothed using Gaussian filter to create a blending mask that ensures smooth transitions at boundaries. This blending mask is applied the occupancy field enhance boundary details. Finally, the marching cubes algorithm generates a 3D mesh for each cluster from the processed occupancy field. These steps allow us to reconstruct instance meshes that support further evaluation of contact predictions.
Figure illustrates the results of our instance meshes.
C
Results
C.1
Visualization
We present additional visualization result samples in Figures D shows further visualization results from our ablation study on architecture. Figure examples comparing contact maps based on 3D space resolution.
E G
and
Additionally, Figure C provides more
16

---

Input Images (Front view)
DMC (Geometry)
Ours (Geometry)
Ours (ID)
GT (Geometry)
Freat view
Back view
Figure D: Extended visualization of our method compared to baseline DMC
43 on Hi4D test split.
Input Images (Foont view)
DMC (Geumetry)
Ours (Geometry)
Ours (ID)
GT (Geometry)
Figure E: Extended visualization of our method compared to baseline DMC 43 on test split of our synthetic dataset.
17

---

Low resolution (128*)
High resalution($122)
Figure F: Example of low resolution contact maps.
Input Images
Geometry Field
ID Field
Contact Field
G.T Geometry
Figure G: Contact Field results of ours in Figure. 3
18

---

Input Images
Geometry Field
ID Field
Contact Field
G.T G Geometry
Figure H: Additional results of our method ContactField in Hi4D test split.
19

---

Input Images
Interaction Coumelay
Input Images
Intecaction Geumelay
Figure I: Failure case.
D Discussion
D.1 Limitation
Despite its effectiveness, our approach introduces certain limitations, such resolution constraints when reconstructing all elements simultaneously, which can affect the finer details the models. Especially, we present failure cases in (d) of Figure Since do not incorporate spatial prior such as SMPL, predicting identity (ID) in extreme poses, such as hugging, becomes challenging. However, shown in (b) from few frames earlier, the prediction is accurate. This suggests that incorporating a temporal module could be a promising direction for future work. Still, experimental results affirm the superiority of our approach over traditional methods, indicating significant potential for future enhancements in complex interaction scenarios and larger group dynamics. Moving forward, we aim refine techniques address these resolution limitations and explore broader applications, further advancing the realism and functionality of 3D human body reconstructions.
E Broader Impacts
This novel implicit field representation for multi-person interaction geometry in 3D spaces has the potential to advance various applications in healthcare, sports, and security by enabling more accurate and detailed reconstructions of human interactions. However, the enhanced ability to capture such interactions also raises important concerns regarding privacy and ethical use. To mitigate these risks, is essential implement robust data protection measures, establish clear ethical guidelines, and ensure compliance with privacy laws.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. We explain motivation and contribution of our work in the main paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We explain the limitation of the proposed method in section D.I of appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Our work is empirical work on interaction geometry for multi-person person interac- tion, which do not requires toehry and proofs. Guidelines:
Justification:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We explain training details in section A of the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer:
[No]
Justification: We do not release the code or data because our work deals with human subject. Although we can not release the code, we will provide the detailed description for our implementation to reproduce all the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
httpl//guuides/ooeeuubiieeeeee www.:obbbsssiooooiooooooo......e for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips.c//pppp
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We explain both training and architecture details in section A the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: We do not present any statistical significance in the table or figure due to strong supervision in reconstruction task. We describe all the details about the experiment for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the computing resources we used for experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp:://brrrpps.ccpppttt/t//t/t
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We explain broader impacts the current work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release the code or data because our work deals with human subject.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We describe all the datasets and models we used in the manuscript.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We describe the details about synthetic dataset introduced in the manuscript.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We the publicly available dataset and synthetic dataset for human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We use the publicly available dataset and synthetic dataset for human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "1XbgtPsBLR", "reviewer": "Reviewer_A26V", "summary": "In this paper, the authors introduce an implicit field representation for multi-person interactive reconstruction. As they said, it can simultaneously reconstruct the occupancy, instance identification (ID) tags, and contact fields. The local-global feature learning methods are used. They also propose a dataset. The result is also good.", "questions": "This is a quite clear paper. The contribution is also good. I wonder the compare results as said in the weakness. Also I want to know the computational cost, including the training/test time and GPU memory.", "limitations": "Yes, the authors discussed the limitations, such as the limitation in finer details.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea, motivation, and model are all clearly presented. The method learns features both locally and globally, and I like the projection operation for the query point as shown in equation (3). Also, the design of the loss function is reasonable. The results also demonstrate the effectiveness of the method. The authors propose a new dataset, which is also a significant contribution.", "weaknesses": "Since I'm not especially well-versed in this area, I did extensive research on relevant works, such as following [1,2,3] etc. However, I noticed a lack of comparative analysis in the experimental part.  I wonder if this work can be compared with those mentioned? If so, what are the results of such comparisons?\n[1]Cha, J., Lee, H., Kim, J., Truong, N. N. B., Yoon, J., & Baek, S. (2024). 3D Reconstruction of Interacting Multi-Person in Clothing from a Single Image. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 5303-5312).\n[2]Mustafa, A., Caliskan, A., Agapito, L., & Hilton, A. (2021). Multi-person implicit reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14474-14483).\n[3]Correia H A, Brito J H. 3D reconstruction of human bodies from single-view and multi-view images: A systematic review[J]. Computer Methods and Programs in Biomedicine, 2023, 239: 107620.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and for recognizing the contributions of our work.\n\nFirst, we will revise our paper to discuss related works including single-view and multi-view settings mentioned in [1, 2, 3]. We emphasize that the mentioned methods [1, 2] primarily focus on single-view reconstruction not multi-view reconstruction. We designed our overall architecture with multi-view settings in mind from the very beginning to handle occlusion challenges in multi-person interaction cases. A multi-view approach fundamentally incorporates triangulation, making it more robust to unseen scenarios and significantly improving frame-to-frame consistency compared to single-view settings.\n\nAdditionally, we understand the importance of providing information on **computational costs**. Here are the details of our computational resources:\n- Training Time: Our model was trained for approximately 2 days with 2 NVIDIA A100 GPUs.\n- Testing Time: Each test instance takes around 60 seconds on the same GPU.\nWe will include these computational cost details in the revised manuscript to provide a comprehensive understanding of our approach.\n\nWe appreciate your thorough research on relevant works and your interest in comparative analysis. If you have any concerns or questions, feel free to leave comments.\n\nReferences\n\n- [1] Cha, J., Lee, H., Kim, J., Truong, N. N. B., Yoon, J., & Baek, S. (2024). 3D Reconstruction of Interacting Multi-Person in Clothing from a Single Image. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 5303-5312).\n\n- [2] Mustafa, A., Caliskan, A., Agapito, L., & Hilton, A. (2021). Multi-person implicit reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14474-14483).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dnLBMTCe8K", "reviewer": "Reviewer_Ar19", "summary": "The paper focuses on the problem of multi-person reconstruction from multi-view images in the face of close interactions (e.g., in cases where are in contact). The objective of this work is to propose a method to reconstruct one mesh per person that is able to capture both fine grained details (e.g., garments, face, hands, etc.) and the contact points between each of these meshes if in contact. The paper introduces a new implicit field representation specially designed for the multi-person setting. This representation allows the reconstruction of occupancy, instance ID (i.e., to which mesh the queried point corresponds), and contact fields at the same time. Aside from the occupancy, the variables for ID and contacts enables to determine which meshes belong to which person so that each mesh can be treated as separate instance. The method facilitates unsupervised estimation of contact points without the need for contact annotations in the training data.\n\nTo achieve the properties mentioned above, the authors propose to enrich the implicit representation by a multi-vew transformer-based feature extraction module to retrieve a mix of local and global features. This enables the modeling of close physical interactions by dense point retrieval in small areas (by exploiting this property of implicit fields to query points at variable resolutions). The work also presents a synthetic dataset containing diverse multi-person interaction scenarios to train the model. \n\nSoTA methods reconstruct multiple people by either: (1) capturing each person’s geometry separately or (2) merging nearby 3D geometries. In this paper, the authors are able to reconstruct multiple bodies at the same time by using the proposed augmented implicit representation.", "questions": "Please, address the questions posed in the **Weaknesses** section.", "limitations": "Yes, the authors include limitations in the supplementary material.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths are the following:\n * The overall quality of the paper is good. The authors clearly identify SoTA limitations, explain them and propose a sound way to deal with these limitations. \n * Overall writing is good as it communicates the ideas clearly, though, authors may need to improve the writing in some parts of the paper to fix formatting issues and typos. (Suggestions are proposed in this review). \n * The method is sound and the proposed techniques used to implement it make sense.", "weaknesses": "### **Ablation missing**\nL217: Authors state that \"bifocal approach of the loss function is crucial\". This statement needs to be backed in the experiments. However, this is not included in the ablation study. I suggest authors include an ablation experiment on the rebuttal, otherwise this is an unsupported claim which weakens the paper by a lot, even to the risk of rejection. \n\n### **More information about the dataset**\nI would like to know how the dataset was created given that it is a synthetic dataset. For example, which software was used to create this. Explain the procedure and add some statistics. Not enough detail is provided here.\n\n### **SMPL close interactions representation**\nL32-33. Can the authors explain why “the need to configure the entire scene on an individual basis complicates the accurate representation of close interactions”? I understand that the SMPL model represents a naked body and thus cannot be used “as is” to reconstruct fine-grained details. However, I don’t understand why having individual instances of SMPL would complicate the representation of close interactions? I think this is not necessarily true. I agree that it does not naturally include a representation for interactions, but this could be added to the SMPL model. Now saying that it complicates the representation may be too much?\n\n### **Baseline models** \nBaseline models: For Baseline models (Sec 4.3). Could you clarify which model MVpose/LVD you use for each dataset? Do you use both for both datasets? LVD only for the synthetic one? Is one of these used by DMC? If not, why not use the one proposed by them? Also why use LVD, is this model not sensitive to occlusion and originally used for only a single person?\n\n### **Minor details**\n* L143: q E R --> q E Z+, it would be more precise as q should not take decimal values, right? It is in a sense a segmentation task. In the case it is R, then is there a threshold to classify each instance?\n* L150: Are the camera parameters K^v estimated or assumed given?\n\n### Observations related to writing\n* Citations typically include the names of all authors. In this case, some citations have all the names, but for most of them, it read \"et. al.\". Also publication conference names are not complete in some cases. For example, L367-368, the conference reads \"IEEE\" which is incomplete. It should say at least \"CVPR\".\n* L27-28. This phrase is badly formulated. First, occlusion does not \"obscure\" a subject. Specifically, saying \"obscures subjects from view\" does not make sense. I would remove that phrase from the last part of this sentence for something like \"...mainly due to occlusion, which complicates the accurate…\"\n*  L32-33. This phrase is not very clear. I understand what it says, but it took me a second read to clarify. Specially, the part that states “configure the entire scene on an individual basis complicates the accurate representation of close interactions” is a bit confusing to me.\n* L65: The name of the module \"transformer-based multi-view local-global\" is a mouthful. I would give a simpler name to this.\n* L81: Related works, when talking about SMPL, I would remove that it does not capture facial expressions as SMPL-X does this.\n* Suggestion, don't use the work necessitates, use needs instead. it is easier to read. Try to avoid fancy wording whenever possible. \n* What is deficit information? Is this a formal term? If not please rephrase this “deficitary information” or something similar. \n### Typos:\n* L102: you could say generative model instead of \"latent vectors\". This gives more clarity.\n* L138: typo. \"using given\", remove \"using\"\n* L150: extracting pixel-aligned features followed by PIFu --> …following PIFu\n* L184: use --> is used\n* L223: Initially, define --> Initially, we define", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insight and agree that including an ablation experiment to support this statement is crucial for the integrity and quality of our paper.\n\n## **Weakness 1:** missing ablation on grouping loss function\nIn response to your suggestion, we are currently conducting an ablation study on each term in the grouping loss function in Eq (12). We will report the performance before the end of the discussion period.\n\n## **Weaknesses 2:** more information about the dataset\nWe present the statistics of the dataset in Figure A. in the attached pdf file. \nInitially, we acquired characters with various ages and interaction motion sequences such as  from Character Creator 4 [1]. Subsequently, we composed scenes featuring multiple characters using Omniverse USD [2] Composer. To facilitate the dataset generation process, we engineered the Kaolin rendering tool [3], incorporating tasks such as multi-person normalization to achieve the desired outputs. As a result, we generated multi-view rendered images, mask images, instance masks, and 3D geometry. Furthermore, we created Normal ground truth maps and joints, although these were not utilized in this paper. \n\n\n\n## **Weakness 3:** SMPL close interactions representation\n> However, I don’t understand why having individual instances of SMPL would complicate the representation of close interactions? I think this is not necessarily true.\n \nIn frameworks like DMC that utilize SMPL, the initial SMPL parameters for individual humans are obtained and then optimized to determine the entire scene. In contrast, our approach derives the human pose and shape for the entire scene in a one-shot manner. We intended to mention that using individual instances of SMPL could complicate the representation of close interactions due to the need for separating parameter optimization for each person, which might require complex coordination. In the paper, we will revise to distinguish between SMPL and the models that use it. \n> I agree that it does not naturally include a representation for interactions, but this could be added to the SMPL model. Now saying that it complicates the representation may be too much?\n\nWe understand your point about the potential benefits of integrating the SMPL model for multi-person interaction geometry. Applying multi-person interaction geometry to the SMPL model is a reasonable approach that can enhance interaction modeling. \nWhile existing methods [4,5,6] optimize SMPL pose estimation for multi-person scenarios, a potential limitation is the need for multiple optimization steps. We believe that leveraging a representation for interactions can effectively optimize the SMPL parameters and address these configuration and optimization challenges. This approach can provide a more accurate and computationally efficient solution for modeling close interactions between multiple individuals. \n\n\n\n## **Weakness 4:** baseline models\nFor the baseline models discussed in Section 4.3, we employed different SMPL acquisition methods tailored to each dataset. Specifically, for the HI4D dataset, we used MVPose, aligning with the experimental setup described in the HI4D paper. In their experiments, DMC was run on the HI4D dataset using MVPose for SMPL acquisition, and we adopted this approach to ensure consistency and comparability.\nHowever, we encountered challenges when using MVPose for the synthetic dataset, as its modules were trained on real data, resulting in less accurate SMPL estimations for synthetic data. To address this, we utilized Learned Vertex Descent (LVD) for the synthetic dataset. LVD is designed to fit SMPL to a 3D human model (3D scan) and has proven to provide more accurate results in this context.\nWhile LVD is originally intended for single-person scenarios and may be sensitive to occlusion, it was chosen for the synthetic dataset due to its superior performance in fitting SMPL to 3D scans, which is crucial for achieving accurate results in our study. To ensure the robustness of our data and fairness in our experiments, we will report the results trained with MVPose on the synthetic data. \n| Model setting         | CD↓| P2S↓ | NC↑|\n|-----------------------|---|----|----|\n| DMC  (w. syn(MVPose)) | 0.805 | 0.489 | 0.771 | \n| DMC  (w. syn(LVD)) | 0.631 | 0.495 | 0.768 | \n| Ours | 0.406 | 0.329 | 0.892 | \n\nWe will revise our paper to include these detailed results and analyses during the rebuttal for providing comprehensive evidence on the proposed framework. \n\n## **Weaknesses 5:** minor details\nThank you for your detailed feedback on our manuscript. We acknowledge the points you raised regarding notation, phrasing, and citation formats. We will address them meticulously in our revisions.\n\nIf you have any concerns or questions, feel free to leave comments.\n\n\nReferences\n\n- [1] Character Creator 4, https://www.reallusion.com/character-creator/\n \n- [2] NVIDIA Omniverse USD Composer, https://docs.omniverse.nvidia.com/composer/latest/index.html\n \n- [3] NVIDIA Kaolin, https://github.com/NVIDIAGameWorks/kaolin \n\n\n- [4] Dong, Junting, et al. \"Fast and robust multi-person 3d pose estimation from multiple views.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\n- [5] Zijian Dong, Jie Song, Xu Chen, Chen Guo, and Otmar Hilliges. Shape-aware multi-person pose estimation from multi-view images. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n- [6] Yuxiang Zhang, Zhe Li, Liang An, Mengcheng Li, Tao Yu, and Yebin Liu. Lightweight multi-person total motion capture using sparse multi-view cameras. In Proceedings of the 17026 IEEE/CVF International Conference on Computer Vision. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZgAMJcCNnO", "reviewer": "Reviewer_Vhig", "summary": "This paper proposes a method to reconstruct close interactions from multi-view images using an implicit representation. The occupancy and ID fields are directly regressed from multi-view images, which can then be used to infer contacts. To fuse multi-view information, a transformer-based module is introduced to consider both local and global features. The authors also built a synthetic dataset to assist in training. The experimental results show that the performance is good.", "questions": "- How can a different number of input views affect reconstruction performance?\n- What is the generalization ability of the proposed method on novel subjects and different camera extrinsics?\n- The ability to address occlusions should be clarified.\n- Why not use contact information provided in Hi4D for contact evaluation?", "limitations": "The limitations and societal impact are discussed in the paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed dataset may be useful for future tasks involving close interaction reconstruction.\n- The qualitative results are good.", "weaknesses": "- The underlying idea is almost the same as DeepMultiCap [39], which also represents interactive humans with pixel-aligned implicit functions and adopts a transformer to fuse multi-view information. Compared to DeepMultiCap, the proposed method does not leverage prior human knowledge and may induce artifacts in occluded body parts. For occluded and interactive body parts, the proposed method cannot sample valid pixel-aligned features. I am confused about the proposed method's ability to address heavy occlusion without any prior human knowledge.\n- The framework is completely implicit like PIFU; however, its generalization ability may not be good due to the high-dimensional output.\n- It seems contacts can also be inferred from the output meshes as pseudo ground-truth contact generation. What is the difference between this approach and the proposed variance estimation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable question. Below are our responses to raised weaknesses and questions.\n## **Weakness 1:** multi-person interaction with SMPL\nThank you for your valuable feedback. In the case of DeepMultiCap (DMC) [1], which uses the SMPL human prior, reconstruction is performed one person at a time. When body parts are occluded, they are masked and included in the input. Using SMPL is particularly helpful in handling occlusions when many body parts are not visible, as it follows the human shape. For areas where body parts are not visible, DMC relies more on the SMPL shape itself than on pixel-aligned features. As the below table shows, the reconstruction performance varies for results from each SMPL method.\n\n| Model setting         | CD↓| P2S↓ | NC↑|\n|-----------------------|---|----|----|\n| DMC  (w. syn(MVPose)) | 0.805 | 0.489 | 0.771 | \n| DMC  (w. syn(LVD)) | 0.631 | 0.495 | 0.768 | \n\nIn extreme poses or close interaction scenarios, estimating SMPL becomes challenging and requires an optimizing process for accurate estimation, which is time-consuming. Additionally, SMPL has limitations in broadly representing the human shape, especially when children are present in our synthetic datasets. SMPL struggles to accurately represent children, making training difficult. \nWe also considered expanding to interactions with entire scene objects, which raised concerns about using SMPL. Therefore, we aimed to handle occlusions by reconstructing the entire geometry from multi-view inputs and predicting the ID field in 3D space using SRT features. By predicting contact based on ID, our method aims to understand multiple people in close interaction scenarios. Additionally, we created and trained our model on synthetic data with numerous instances of self-occlusion and multi-person interaction using benchmark datasets to ensure robust SRT feature extraction.\n\n- [1] Yang Zheng, Ruizhi Shao, Yuxiang Zhang, Tao Yu, Zerong Zheng, Qionghai Dai, and Yebin Liu. DeepMultiCap: Performance capture of multiple characters using sparse multiview cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n\n## **Weakness 3:** contact information from the output meshes\nThank you for your insightful comments regarding the generation of pseudo ground-truth contacts from output meshes and the proposed variance estimation method for contact detection. Contact label generation relies on geometric proximity and surface identifiers to infer contacts directly from the mesh, while Estimated Contact Fields use statistical variance to detect contacts within the 3D space.\nTo address your comment, we have inferred contacts from the output meshes using the contact label generation approach and measured the precision of our contact predictions. Examples of instance meshes are presented in Figure B in the attached PDF file.\n\n|         | CP $( \\\\delta = 0.05 $)↑ | CP ($ \\\\delta = 0.075 $)↑ |\n|-----------|---|---|\n| output meshes (requested) | 0.518 | 0.621 | \n| variation estimation (ours) | 0.629 | 0.670 | \n\n\n## **Question 1:** the performance of different number of input views\nWe will report the performance of our model with smaller views before the end of discussion period.\n\n## **Weakness 2 and Question 2:** generalization ability on novel object or different extrinsic camera parameter\nTo thoroughly evaluate this aspect, we conducted two experiments using our pre-trained model. These experiments included zoom-in and zoom-out tests with the Hi4D dataset, as well as tests on synthetic data featuring four individuals performing extreme poses, such as breakdancing, introducing novel postures and configurations unseen during training with new camera settings. Qualitative results are provided in Figures C and D of the attached PDF. \nDuring the zoom-in and zoom-out tests on Hi4D, we adjusted the images and camera parameters accordingly. In the zoom-out scenario, the person's size in the image decreased, leading to a loss of detail in feature extraction. \nHowever, our implicit fields prevent significant performance degradation by taking the 3D space bounding box around the person's center, thereby preserving the geometrical results. The qualitative results, shown in Figure C. indicate that the model can handle previously unseen zoom-in and zoom-out scenarios. Notably, the model performed better during zoom-out than zoom-in for ID field prediction, as it relies on extracting features from the entire scene.\n\n## **Question 3:** robustness on occlusion \nOur approach addresses occlusions by presenting interaction geometry. We estimate complete geometries from multiple views and use SRT features to predict IDs and contact regions in 3D space, even in occluded areas. By predicting contact regions based on IDs, our model handles occlusions in close interaction scenarios as shown in the first row of Figure B in the attached PDF file. \nAdditionally, we created synthetic data with numerous multi-person interactions and trained our model using this data alongside benchmark datasets to enhance robustness in extracting SRT features. \n\n## **Question 4:** Why not using contact information of Hi4D\nTo the best of our knowledge, while the Hi4D dataset provides SMPL contact ground truth (G.T.) information, we could not find any contact G.T. information specifically for the 3D scans in the Hi4D dataset.\nAs a result, we generated a pseudo ground truth using the 3D instance mesh provided by Hi4D and used it for our measurements, as detailed in the paper. We appreciate your question and will include a clarification on this point in the revised manuscript to ensure transparency.\nThere may be misunderstandings in our responses, so it would be great if we could continue the discussion regarding any unclear or intriguing parts of our answers or any unresolved questions.\n\nIf you have any concerns or questions, feel free to leave comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "037mXHalYN", "reviewer": "Reviewer_N9qi", "summary": "A novel implicit field representation is designed for multi-person geometry modeling, which manages to estimate the occupancy, identity, and geometry simultaneously. Moreover, to alleviate the occlusion issue, an additional 3D scene representation module is designed. A synthetic dataset with multi-view multi-human interaction is developed. Experiments show the superiority of the proposed method.", "questions": "- The aspect ratio for the data sample videos seems not right. \n\n- In inference, how are the different persons separated by the predicted ID values in detail? Is the number of existing human instances required as input? \n\n- How is the local neighborhood \\mathcal{N}(v) decided in L233?", "limitations": "The limitations have been discussed by the authors.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Estimating the contact fields by the variance of the predicted ID field is an interesting idea and has shown effectiveness. \n\n- The performance is impressive both quantitatively and qualitatively.", "weaknesses": "- More details on the proposed synthetic dataset should be provided, including the data source, the synthesis and annotation protocol, and the data scale.\n\n- Some interested hyper-parameters are not specified, including the contact deviation threshold \\tau_c in L236.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback on the synthetic dataset and our experiments. Below are our responses to weaknesses and questions in your comments.\n\n## **Weakness 1:** details about datasets\nWe outline the relevant information about the dataset below.\n\n**Data Source**:\nInitially, we acquired characters with various ages and interaction motion sequences from Character Creator 4 [1].\n\n**Synthesis and Annotation Protocol**:\nSubsequently, we composed scenes featuring multiple characters using Omniverse USD Composer [2]. To facilitate the dataset generation process, we engineered the Kaolin rendering tool[2], incorporating tasks such as multi-person normalization to achieve the desired outputs. This process enabled us to generate multi-view rendered images, mask images, instance masks, and 3D geometry. Additionally, we created normal ground truth maps and joints, although these were not utilized in this paper.\n\n**Data Scale**:\nThe dataset includes 49 characters (26 female, 23 male) with 50 motion sequences across 43 scene configurations, totaling approximately 25,000 frames. For a detailed breakdown, please refer to Figure A in the attached PDF file.\nWe hope this detailed information will provide a clearer understanding of our synthetic dataset and its role in evaluating our proposed method.\n\n## **Weakness 2:** detailed hyper-parameters\nThe hyper-parameters are as follows: $\\tau_{c}=0.25$ in L236, $\\omega_{s} = 1, \\omega_{\\text{contra}} = 0.1 \\omega_{\\text{group}}=0.1$ in equation (9). We will ensure these details are included in the revised version of the paper. Thank you for your careful review and pointing out the missing hyper-parameters.\n \n## **Question 1:** aspect ratio \nThank you for your observation regarding the aspect ratio of the data sample videos.   The aspect ratio of figures in the paper is aligned with benchmark datasets such as Hi4D. However, it appears that the sample video had the contact field rendered with a different aspect ratio compared to the other results. We have re-adjusted the aspect ratio and created a new video. Refer to adjusted frame examples in Figure E. of the attached PDF.\n\n\n##  **Question 2**\nHere is a detailed explanation of how different people are distinguished using predicted ID values:\n\n**1. Region Identification**: During inference, the algorithm identifies and marks regions of interest based on the occupancy field, excluding background regions. This step generates both an ID field and a contact field. \n\n**2. Clustering**: The normalized ID values within these regions are processed using a clustering algorithm, specifically K-Means. This algorithm groups the data into unique clusters, each representing a different person, and includes the contact regions for all individuals.\n\n**3. Segmentation**: After clustering, each cluster is isolated with a binary mask. A binary mask is then created and smoothed with a Gaussian filter to form a blending mask, ensuring gradual transitions at boundaries. This blending mask is applied to the occupancy field to enhance boundary details. Finally, the marching cubes algorithm generates a 3D mesh model for each cluster from the processed occupancy field.\n\nInstance mesh visualization results are presented in Figure B. of the attached pdf file.\n\n## **Question 3:** local neighborhood\nThe local neighborhood $N(v)$ is defined as the set of points within a specified radius (contact threshold) around the point $v$, excluding $v$ itself, and is determined using a KDTree [4] for efficient querying. The criteria for marking a point as a contact is based on the presence of neighboring points with differing surface identifiers.\n\nIf you have any concerns or questions, feel free to leave comments. \n\nReferences\n\n- [1] Character Creator 4, https://www.reallusion.com/character-creator/ \n\n- [2] NVIDIA Omniverse USD Composer, https://docs.omniverse.nvidia.com/composer/latest/index.html\n\n- [3] NVIDIA Kaolin, https://github.com/NVIDIAGameWorks/kaolin \n\n- [4] Bentley, Jon Louis. \"Multidimensional binary search trees used for associative searching.\" Communications of the ACM. 1975.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their thorough constructive comments on our paper.  We earnestly responded to your concerns; please see the respective comments. Before answering your questions and concerns, we would like to highlight our contributions by quoting your comments.

### **Strengths of the proposed work**
- **Clear presentation**: Multiple reviewers ( **reviewer A26V**, **reviewer Ar19** ) appreciated the clear presentation of ideas, motivations , and models.
- **Innovative ideas and effectiveness**: The innovative methods, such as learning both local and global features and the projection operation for query points, were noted positively (**reviewer A26V**, **reviewer N9qi**). The effectiveness of these methods was demonstrated through strong quantitative and qualitative results (**reviewerA26V**, **reviewer N9qi**).
- **Contribution of new dataset**: The introduction of a new dataset was seen as a contribution by reviewers (**reviewer A26V**, **reviewer Vhig**), indicating its potential usefulness for future research.
### **Weaknesses of the proposed work and raised questions**
- **More information about the dataset** :  In response to **reviewer Ar19** and **reviewer N9qi**, we have provided more comprehensive details regarding the synthetic dataset creation and a detailed explanation of the proposed synthetic dataset and illustrated the data statistics in Figure A. of the attached PDF file. 
- **Comparison to SMPL with close interaction**:  **reviewer Ar19** and **reviewer Vhig**  mentioned about SMPL closed interactions representation compared to our method. In frameworks like DMC that utilize SMPL, the initial SMPL parameters for individual humans are obtained and then optimized to determine the entire scene. In contrast, our approach derives the human pose and shape for the entire scene in a one-shot manner. 
- **Generalization ability on novel subjects and camera extrinsic parameters**:  These experiments included zoom-in and zoom-out tests with the Hi4D dataset and tests on synthetic data with four individuals performing extreme poses like breakdancing, introducing novel postures and configurations with new camera settings, as shown in Figures C and D of the attached PDF.
- **Computational costs** Requested by **reviewer A26V**, we report the computation costs of our model. We train our model on 2 days with 2 NVIDIA A100 and test 60 seconds for each instance.
- **More experiments**:  **reviewer Ar19** mentioned the need for an ablation study, and **reviewer Vhig** requested a smaller view study. . We are currently training our model for the ablation study and both our model and the baseline model for the smaller view experiments. We will report the results before the end of the author-discussion period. 


To accommodate your requests, we have attached an **one-page PDF** for visualization of newly conducted experiments.
If you have any concerns or questions, feel free to leave comments.

Meta-review: Thank you all for the very detailed reviews, rebuttal, and extensive time spent in the discussion. Three out of the four reviewers are positive, with the last one having most of their concerns addressed. The remaining point of reviewer Vhig is the overlap in persons between train/test sets, but this is following the existing protocols/practices. The paper presents a solid approach for representing multi-persion interaction. Please incorporate the additional results and explanation given throughout the rebuttal and discussion phase in the final version.
Decision: Accept (poster)