Humans in Kitchens: A Dataset for Multi-Person Human Motion Forecasting with Scene Context
Authors: Julian Alexander Tanke
Oh-Hun Kwon
Felix Benjamin Mueller
Andreas Doering
Juergen Gall
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: 3D Human Motion
3D scene environment
TL;DR: Multi-person 3D Human Motion Dataset with annotated 3D environment

Abstract: Forecasting human motion of multiple persons is very challenging. It requires to model the interactions between humans and the interactions with objects and the environment. For example, a person might want to make a coffee, but if the coffee machine is already occupied the person will have
to wait. These complex relations between scene geometry and persons arise
constantly in our daily lives, and models that wish to accurately forecast
human behavior will have to take them into consideration. To facilitate research in this direction, we propose Humans in Kitchens, a
large-scale multi-person human motion dataset with annotated 3D human poses, scene geometry and activities per person and frame.
Our dataset consists of over 7.3h recorded data of up to 16 persons at the same time in four kitchen scenes, with more than 4M annotated human poses, represented by a parametric 3D body model. In addition, dynamic scene geometry and objects like chair or cupboard are annotated per frame. As first benchmarks, we propose two protocols for short-term and long-term human motion forecasting.

Humans in Kitchens: A Dataset for Multi-Person Human Motion Forecasting with Scene Context
Julian Tanke"
Oh-Hun Kwon nakwon@uni-bome de
tanke@iai uni-bonn... de
Felix B Mueller
Andreas Doering" doering0iai. uni uni-bonn.dom de
felix. benj jeinamii.oo muelleree weleerrrniiiiee Suniunonnnnn..... de
Juergen Gall" :
gall@iai. uni-bonn.de
Abstract
Forecasting human motion of multiple persons very challenging. It requires to model the interactions between humans and the interactions with objects and the environment. For example, person might want make coffee, but if the coffee machine is already occupied the person will have to wait. These complex relations between scene geometry and persons arise constantly in our daily lives, and models that wish to accurately forecast human behavior will have to take them into consideration. To facilitate research in this direction, we propose Humans in Kitchens, large-scale multi-person human motion dataset with annotated 3D human poses, scene geometry and activities per person and frame. Our dataset consists of over 7.3h recorded data of up to 16 persons the same time in four kitchen scenes, with more than 4M annotated human poses, represented by a parametric 3D body model. In addition, dynamic scene geometry and objects like chair or cupboard are annotated per frame. As first benchmarks, we propose two protocols for short-term and long-term human motion forecasting.
1 Introduction
Understanding and anticipating human motion within groups is very challenging and essential in the context of socially-compliant autonomous robots [1, 2, 3, 4, 5, 6, 7, 8), as they must possess the ability to understand and respond appropriately human behavior. Moreover, this topic has relevance the fields neuroscience and social sciences [9, 10. 11], as it enables the development of computational models that explore the perception of others' behavior and its influence on one's own behavior. For example, imagine group persons sitting on sofa and another person walking towards it, would expect the person sit down persons in front a whiteboard are expected to discuss their ideas. However, is more plausible that only one of them writes onto the whiteboard while the other observes.
unoccupied seat the sofa. Similarly, two
Capturing social interactions and interactions with the environment necessitates a large dataset for effective training and evaluation. Such dataset must possess three essential characteristics: (a) it should encompass natural interactions between multiple individuals recorded in real environment,
University of Bonn
Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS LLamar Institute for Machine Learning and Artificial Intelligence
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

A
R
Figure 1: Humans in Kitchens consists of 7.3h captured human poses of multiple persons in four different kitchen environments A, B, C and D.
(b) it should have annotated scene geometry to account for interactions with the scene, and (c) should include annotated per-person action labels to balance the evaluation and avoid a strong bias towards simple activities like standing, walking, and sitting. Currently, such dataset with 3D human poses does not exist as shown in Tab. The largest multi-person human motion dataset is Panoptic Studio [12]. The dataset, however, has been recorded in a studio. Although the persons interact, they mainly stand due the small recording area. also does not include a real environment and the subjects need to act in an unfamiliar environment with many cameras and light sources, which can induce behaviour that differs from real-life behaviour.
In order to address these issues, we propose Humans in Kitchens, a large-scale multi-person 3D human motion dataset with annotated scene geometry and per-person activities. Our dataset consists of more than 4M unique poses 90 individuals in total. We recorded persons four real kitchen for over 7.3h. Each of the four kitchen sequences was continuously recorded for 1.5h to 1.9h. Persons could freely enter or leave the scene and received minimal instructions, resulting in a very natural behavior and interactions. For each scene, annotated objects that people may interact with, such as sinks, dishwashers, chairs, or whiteboards, and objects that determine the geometry of the scene, such as walls. Some objects, such as chairs and kettles, are annotated per frame as they may be moved around the scene. The objects. however, are only coarsely annotated by 3D boxes and cylinders. While such an annotation allows to learn based on scene context where and how activities are performed, the dataset cannot be used to model fine-grained human-object interactions like rotating the knob of the microwave or grasping knife. The scenes span between 38m² to 80m?, which is much larger than the 19m2 of Panoptic Studio [12]. The maximum number of individuals in the scene at the same time 16, twice the maximum number of persons in Panoptic Studio. For each person, we annotate their frame-wise activity, such as walking, sitting, writing whiteboard making coffee. We represent humans by the SMPL [13] body model, which includes the 3D skeleton pose. We belief that Humans in Kitchens will contribute to advance multi-person person human motion forecasting as well as modeling scene context for social behaviour understanding and anticipation.
We provide details on the acquisition and annotation of the dataset, dataset statistics, and evaluate state-of-the-an methods for multi-person human motion forecasting. We further discuss limitations and potential risks of the dataset.
2 Related Datasets
We present the most related datasets with one or multiple 3D human poses Tab. and briefly discuss them.

---

Table 1: Comparison of various datasets with 3D human poses. The real data column specifies if a dataset contains real, synthetic, or partially synthetic human motion. The real setting column specifies if the recording was done in a controlled studio environment or in a real-world scene. The SMPL. column determines whether a dataset provides SMPI. [13] poses. The column max # P) specifies the maximum number of persons at the same time in a scene while columns activities and scene determine if the dataset contains per-frame annotations of activities or scene geometry. Numbers in activities indicate the number of different per- per-frame activities.
Datasel
real data
real setling
SMPL.
max (gfe P)
total time
activilies
framerate
AIST [14)
yes
yos yes 100
80
5.2h
B0
AMASS [15]
yes
n0
4th
80
50
BEHAVE [16]
8.5min
1081:
CHAIR [17]
yes
17.3h
30H2
CHICO [18]
yes
yes
3,77h
BO
yos
25Hz
GIMO [19]
yes
yes
yes
1.2h
80
10H
GTA-IM (20)
n0
9.2h
DO
30H2
Human3.6M 6M (21]
299
SUHZ
Hamanise (22)
no
yes
5,55h
yos (Iaggagg)
yos
MoGize [23]
yes
n0
3h
BO
120HZ
PROX [34)
55min
SAMP (25)
pirtial
yes
100min
3DPW [26]
yes
ye5
yes
Femin
10
60
CHESD [27]
ye5
30
40min
D0
200Hz
CMU Mocap [28]
9,75h
60L12 12011c
Plncgttic [29]
yes
80
5.5h
20 97H2
EgoBody (30)
yes
yes
yes
2h
B0
yes
2011
ExPI [31]
110
30
20min
250
Haggling dataset [32]
yos
97
MuPoTSS (33)
yes
yes
<4.4min
BO
NTU-RGB-D 120 [34]
yes
110
300
63min
yes (120)
300
RICH [35]
yes
2.7h
DO
GUH2
Homons Kitchens
yes
25Hz
Single-Person Datasets: AMASS [15] unifies several 3D human motion datasets using SMPL [13]. In total, AMASS contains over 40 hours of motion capture recordings. While the underlying articulated model ensures high quality motion, AMASS only contains recordings of a single person and thus no human interactions. also does not contain scene context information or per-frame action annotations. BEHAVE [16] contains fine-grained human-object interactions but it is recorded at only 10Hz and very small (8.5 minutes). Another human-object interaction dataset is CHICO [18], which contains 3.77 hours of recording. In contrast to BEHAVE, the human poses represented by 3D keypoints instcad of SMPL [13] body poses. Human3 6M [21] contains around 900k high- quality human 3D poses. The motion sequences, however, are unrealistic since the actors pantomime activities without objects except of sitting on a chair. Other objects or 3D geometry are missing. MoGaze [23] contains 3 hours of human-object interactions with annotated 3D geometry. PROX [24] provides very high-quality human-object interactions with static objects such as chairs, beds and sofas and utilizes SMPL as body model. POSA [36] extends this with generative model. The GTA Indoor Motion dataset (GTA-IM) [20] utilizes the GTA engine to produce human-Scene interactions in indoor environments for human motion forecasting. The motion in this dataset, however, is synthesized and looks unrealistic and clumsy. Similarly, SynBody [37] utilizes pre-rendered scenes and synthetic humans to produce diverse sequences. SAMP [25] is human-scene interaction dataset containing 7 real objects, such as sofas and armchairs, and various human interactions with those. An extensive augmentation pipeline is used to extend the dataset with a greater variety of human-object interactions. AIST+ [14] contains 30 subjects dancing to music sequences. GIMO [19] contains a single person interacting with a static, high- ligh-quality mesh, where data is provided in the form of an egocentric viewpoint. The dataset was explicitly designed for motion forecasting, where a person walks into scene with the intent to interact with an object. Our dataset contains multiple persons, 5 times more frames, and moving objects. Humanise [22] is a large-scale synthetic homan-object interaction dataset that leverages existing datasets in human motion (AMASS) and 3D indoor scenes [38]. Similar to GIMO, the scenes are static. Furthermore, the person-object interactions are synthetic and do not include real person-object interactions. CHAIR [17] is a large-scale human-chair interaction dataset that contains large variation of chairs. In contrast to these datasets, Humans in Kitchens contains multiple persons.
Multi-Person Datasets: 3DPW [26] contains 2 persons per scene, but the 3D joint locations were obtained from moving cameras, resulting in unrealistie sliding. In total, this dataset consists of 14 minutes of recording. CMU-Mocap [28] is a high-quality motion capture dataset of I to 2 persons. While some of the single-person sequences contain scene interaction, the scenes are not annotated. The dataset does not contain accurate action labels, but the sequences can be searched based on
3

---

Figure 2: Overview of each of the four kitchens. Each kitchen contains sofas and chairs (red tables (blue m) and at least one whiteboard (orange ). fridge (green coffee machine (yellow and sink (dark blue Best viewed using zoom PDF viewer.
the video descriptions. CHI3D [27] consists of 40 minutes of two-person interactions, including close interactions such as touching. Human poses are represented as 3D skeletons and as SMPL-X models [39]. CMU Panoptic Studio [12. 29] is a large-scale 3D human motion dataset featuring 1 to 8 persons in various scenes. The human poses are represented using 3D COCO keypoints. The dataset size is 5.5h, but lacks labeled scene geometry and per person action annotations. Due the studio recording, the range of motion is limited. MuPoTSS-3 [33] is a test set for 3D human pose estimation. contains up to 3 persons NTU-RGB+D [40, 34] is an action recognition dataset containing more than hour of recording of one or two persons. Human bodies represented 3D skeletons and per-frame activities annotated. In contrast to these datasets, Humans in Kitchens has been captured in a real environment, includes annotated scene context and frame-wiss activities per person. EgoBody [30] contains sequences of pairwise social interactions with real scene geometry. In contrast to our work, the scene geometry is only statically annotated and social interactions are less natural, as one actor wears virtual headset. RICH [35] captures human-object interactions by defining pseudo-contact labels on the body mesh. Unlike PROX [24), RICH contains mostly outdoor scenes of around 60m2 and provides more accurate SMPL.-X estimates. Some of the scenes contain two humans who interact with each other from a distance like throwing a ball.
scene but contains only 4.4 minutes of recording.
3 Humans in Kitchens
In order to obtain dataset that one hand contains realistic behaviour of multiple interacting persons and on the other hand is GDPR conform, which includes the informed consent of each subject in the dataset, of asking subjects to perform certain motions or games in a recording studio, we collected data in four real office kitchens for duration between 1.5h and 1.9h. Persons were allowed enter and leave the scene and persons that did not want to participate were asked to use another kitchen during the recording session. Despite of the informed consent sheet, only subset of the participants received minimal instructions as we will discuss in Sec. 3.1.1. The dataset acquisition and annotation process will be described in Sec. 3.1 and See. 3.2.
followed different approach than previous datasets with 3D human poses. Instead
3.1 Dataset Acquisition
For the data acquisition, 11-12 calibrated cameras that were synchronized via the audio signal have been used. The recording has been performed in four different office kitchens on different days. The layout of the kitchens is shown in Fig. While the hardware setup is described in [41], corresponding repository [41] contains only the raw data where the estimated 3D human poses are very noisy due to severe occlusions and limited view of each camera. The raw data itself can thus not be used for training evaluation. In Sec. 3.1.2, we describe how the raw data has been manually

---

Legend:
(a) Individual head
Automated processing
3d person locations
Human annotation
Data
(b) Occlusion annotation
(f) OpenPose to SMPL
(c) 3d pose estimation
poees (OpenPose)
poses (SMPL)
(g) fill-in masked SMPL
(d) 3d pose annotation
Occlusion masks
(e) Occlusion annotation
Overview of the 3D human pose annotation process. First, (a) the heads of individual persons are annotated each frame, represented as 3D point in global coordinates and unique identity. Given the person annotations, automatically extract 3D human poses (c). We manually correct 3D human poses (d) or mask them (e) if even the annotators cannot correct them. We estimate SMPL parameters (f) and inpaint masked regions (g) to obtain for each pose a full SMPL pose representation. Black boxes represent automated processes without human intervention, gray boxes represent human annotation processes and blue boxes represent data.
Figure 3:
annotate occlusions for each person and frame (b) and
annotated to obtain the Humans in Kitchens dataset for multi multi-person person human motion forecasting with scene context.
3.1.1 Behavior Protocol
For each of the four recordings, cake has been provided to attract participants to the kitchen. To facilitate behavior natural as possible, we provided only minimal instructions to 10 persons each recording, where they were asked to randomly perform 3 of the following activities at any time and in any order:
Make coffee: Prepare coffee and drink it; Make tea: Use the kettle to prepare a tea and drink it; Eat cake; Take a slice of the cake and cat it; Eat fruit: Eat some of the provided fruits; Drink water: Drink water from the tap; Explain on Whiteboard: Explain topic of your choice on the whiteboard; Use Laptop: Work
laptop: Use Microwave: Use the microwave heat milk for coffee; Read paper: Read a paper; Make phone call: Make phone call; Clean dish: Clean your dish(es) in the sink; Place in dishwasher: Put used dishes the dishwasher.
While 10 persons were instructed, the other persons present in the scene were not instructed to perform any of the above mentioned activities. However, each person allowed to perform any activity, e.g., anyone could make a coffee, eat cake or clean dishes.
3.1.2 Pose and Activity Annotation
We annotated the 3D human poses of each person in each frame where each person has unique identity through the sequence. To extract 3D human poses in our very challenging environment, we annotated the 3D poses in five phases:
Manual nose annotations: We first manually annotated each individual each frame at their nose in the 3D scene, using custom annotation tool. This also allowed to re-identify persons who left the scene but later returned to the recording. We verified the correct annotations in a second pass where additionally the 82 activities where annotated per frame per person. In total, annotating this phase took around 2,000 person hours.
Automated human pose estimation: We ran an off- the-shelf 3D human pose estimation method [42] extract the 3D human poses from the multiple cameras. We match the estimated 3D poses to closest manually annotated nose and drop all leftover poses. If 3D nose annotation is not matched to an estimated 3D pose, we linearly interpolate between the previous and the next frame. The extracted 3D human poses represented 3D skeletons using the OpenPose keypoints [43].
5

---

Table 2: Dataset statisties for the four kitchen environments A, B, C and
B
Rotal
(1893991448)
(1..009
(1.5392(1.95)
(7,,244
(7.33)
annonated poses
$73,253
1,132,422
780
1,415.199
4,079, 2079,244400 (44.76))
meon persons/ frume
4.42
6.32
5.17
7.97
modian persons/ / frame
6
5
mas. persons/
individuals
18
32
16
24
90
surface
76.33m
57.22m 2
38, 2
80.40m
scene objects
40
29
Manual occlusion masking and human pose correction: The automated pose estimation method fails in heavily occluded scenes, requiring us to manually correct the 3D skeletons those frames. We manually annotated and corrected 20,000 3D poses, around 0.5% of the dataset. Additionally, we manually annotated occlusion masks for head, upper body and lower body where even human annotators were not able determine the correct 3D joint positions. Differentiating between head, upper and lower body is a compromise between accuracy and annotation speed, as often only certain parts of a person, e-g., the legs, were occluded. In total, 6% of the poses have at least one body part masked, This annotation phase required 600 person hours.
We adapt off-the-shell optimization framework* to extract SMPL parameters from the 3D OpenPose keypoints that have been annotated the previous steps. We extract the SMPL parameters for all poses, even the masked once. We can do this as the masked poses still represent valid 3D poses as we just interpolate between the known frames. We determine the shape for each actor beforehand as we aware of their height, weight and general body circumferences. We use neutral body for every actor.
Fitting SMPL:
SMPL Inpainting: We utilize an unconditioned human motion diffusion model (MDM) [44], trained on AMASS [15], to inpaint the masked poses. For this, we subsample AMASS to 25Hz to match the frame-rate of our dataset. During inference of the diffusion model, we replace the unmasked keypoints by the annotated keypoints at each diffusion step. In this way, the annotated and verified 3D keypoints remain unchanged, but the occluded and masked parameters will be filled by the diffusion model.
Figure 3 provides an overview of our pose annotation process and we provide more details in the supplementary material.
3.1.3 Scene Annotation
We annotate the scene geometry either as 3D box or as cylinder, where we annotate trash bins, stools and circular tables cylinders and everything else as box. In each of the four kitchens, annotate the following 13 objects: Whiteboard, Microwave, Kettle, Coffee Machine, Table (sofa table, bar table, kitchen table), Sittable (sofas, arm chairs, chairs, and stools), Cupboard (floor and hanging cupboards), Occluder (walls and pillars), Dishwasher, Drawer, Sink, Trash, and which marks the boundary of the visible area. The objects in the scene are annotated per frame since the objects can move during the recording. Fig. 2 shows four examples.
3.2 Statistics
Our dataset is a large-scale multi-person motion dataset, recorded at 25Hz, with over 4M individual 3D human poses of 90 individuals and over 650k frames, total of 7.3h of recording, as summarized in Tab. 2. Compared to other datasets with 3D human poses, it contains more numbers of persons in the scene and more diverse activities recorded in real environment as summarized in Tab. 1. Furthermore, the dataset not only includes the context of static environment, but also moving objects that have been annotated. Each pose further labeled with one or multiple activities out of 82 activity classes, for example, sitting in chair, writing on a whiteboard, or washing hands". Scene geometry annotated per frame as either a 3D box or as 3D cylinder as well as with one out of 13 object classes, e.g., coffee machine, table, whiteboard. The dataset was recorded in 4 kitchens, A, B, and D, with common scene geometry such as coffee machines, chairs and whiteboards, but with
'nttpa:/ htt///gitttt thtbb com/Dou- SonnDoa--Yiing/Poan Viming/Pose, sassee_tt__M____ 'A full list is provided in the supplementary material

---

A
B
#person scene
1.4
lotall nelking
Marding
atting
C
D
éperson scone
minutas
minites
Figure 4: Number of persons that are Walking, Standing or Sitting a frame. The x-axis represents the elapsed time in minutes while the y-axis represents the number of person that perform the corresponding activity. The black curve the total number of persons frame.
Silhee
Using sink
000
Figure 5: Occurrence map for the activities Walking, Sitting and Using sink. The maps generated by plotting the location of the root joint of each person in the entire recording when persons perform the corresponding activity. For the last activity, Using sink, highlight the location with red circle as the activity only occurs close to the sink (dark blue).
different room layouts, as can be seen in Fig. 2. The number of annotated objects varies between 29 and 50. Each kitchen sequence has been recorded continuously, taking between 1.5h (A) to 2h (B, C, D). While there are average between 4.42 and 7.97 persons same time visible scene, number of persons varies largely during sequence since persons enter and leave the scene as shown in Fig. 4.
From the 82 annotated activities, we define 5 as posture activities: Walking, Sitring, Standing, Leaning and Kneeling. each point in time, person exhibits one of the 5 postures. We differentiate between Leaning and Standing by defining that person leans if the person's 's weight is supported by scene object, e.g., cupboard. Kneeling very rare and is only briefly observed in two kitchens: we decided to annotate it for completeness as it would not fit any of the four other postures. The postures greatly vary frequency as we show in Tab. 3. The most common postures are Sitting and Standing, which is expected from natural social human interactions. In Fig. 4, we plot the three most common postures, Walking, Standing and Sitting over time. We observe that in B and D there is time window at the end of the recording where most persons sit. This is not observed in A and C. This shows that
7

---

Table 3: Number of frames and persons per dataset for the posture activities Walking, Sitting, Standing. Leaning and Kneeling.
Action
A
B
C
D
Sum
Walking
39,454
73,051
79,195
119,230
310,930
Sittng
265,521
314,791
263,652
631,673
1,475,637 (16.39h)
Standing
209.525
635,835
464,940
569,712
1,880,012 (20.89h)
Leaning
25,491
46,538
56,846
44,685
173, 173,560 5.60 (1.9h)
Kneeling 0
702
0
50
752 (30s)
the distribution even of the basic posture activities varies over time and from scene to scene. In Fig. 5, we show a bird eye view for all four kitchens and plot where three example activities occur. As expected, Walking covers almost the entire scene while object-specific interactions with the scene such as Sitring and Using sink are localized at the corresponding scene objects. Although chairs move in the scene, they are not moved to completely different locations since the kitchens offer sufficient chairs.
3.3 License and Consent
The dataset and API are free to download and can be used for non-commercial purposes. The API is under MIT-License while the dataset utilizes a custom license. All subjects signed forms consenting that recordings of them derived of them can be used for non-commereial research purposes. The recording has been approved by the ethical review committee of the University of Bonn. In contrast to the data [41], Humans in Kitchens does not contain personally identifiable information. The raw data is also accessible', but requires to sign a license agreement and is subject to export regulations. Note that the raw data is not required for using Humans in Kitchens. The data does not contain any offensive content.
4 Experiments
We evaluate various state-of-the-art methods on the human motion forecasting task of our dataset. More precisely, the goal is to learn function that takes as input human pose sequence X (xt=) xh ... x of n persons, where x!: E (2x(3))3) represents 3D motion sequence of t frames for person i, and forecasts the future motion for all n persons in global coordinates:
X s(x!
I+I E (T(T=t)xxx(2xxx
(1)
As our dataset contains natural behavior over very long time, we select 2 interesting motion activities, namely Walking and Sitting down, and 4 interesting interactions with the objects Whitehoard, Sink, Cupboard, and Coffee Machine for evaluation. Note that most of these actions involve social interactions since the persons sit in groups and discuss at the whiteboard. For training, we use the kitchens A, B and C and we evaluate on D, which is the largest among the four kitchens. For the hujent interactions, we select the last observed frame t as the first frame of the annotated action, while for the motion activities are already observed. Overall, we sample all occurrences of the given activity in the test set We only evaluate the forecast motion of the person that performs the activity and not for any other person in the scene. This allows to report accuracy per activity, but it still requires to model the context of the other persons. We consider two distinct protocols: short-term and long-term motion forecasting. In the short-term protocol, we employ the widely used Mean Per Joint Positional Error (MPJPE) metric [45] measure the positional disparity between the predicted motion and the ground truth. For the long-term protocol, we use the Normalized Directional Motion Similarity Score (NDMS) [46], which effectively assesses the quality of motion sequences of any given length.
select frame t 10 such that few frames of the motion
Baseline methods: We evaluate recent single-perse person human motion forecasting methods, namely siMLPe [47], CHICO [18), pgbig [48] and Repeats-ppttttttee [49], and the multi-persod forecast- ing method Multi- Range Transformers (MRT) [50]. While the single- pegge-eersonn approaches forecast the
htttps///gthhuu..omm//uaaaa////s con/jutankeehikk
'nttpa://github. http://gtttubb..obbbbobeee com/bonn www..//oonaaciiiityyoooooeeeeeee activity bacaapp/boon_aaa caann__ttvvttymmm
'More details provided in the supplementary material
8

---

Table 4: MPJPE L. in dm. Methods denoted with forecast all persons at the same time.
frame
15
25
15
5
15
waakking
150)
0.24
0.29
0.28
0.51
1:19
[47]
0.97
0.24
0.17
0.3
0.20
0.86
0.88
0.36
0.27
0.19
0.33
0:65
0.33
0.55
pabic
0.85
0.87
0.16
0.20
0.20
1hsRep
0,83
0.30
0.19
0.2
0.16
0.37
Table 5: NDMS 1. Methods denoted with forecast all persons at the same time.
frame
25
100
200
250
25
100
200
250
25
100
200
250
walkinn
sitling down
whitehoard
MRTT (50)
0.34
0.38
0.33
0.81
0.26
0.10
0.80
0.11
0.10
sUMIPP [47]
0.87
0.41
01244
0.20
0.19
0.87
0.57
0.22
0.19
0.18
0.88
0.35
0.21
0.18
0.17
CHICO [18]
0.84
0.40
0.25
0.20
0.19
0.85
0.36
0.22
0.19
0.18
0.36
0.35
0.22
0.18
0.17
peligg
0.43
0.25
0.21
0.87
0.38
0.19
0.87
0.23
0.20
0.19
HorRop [49]
0.46
0.21
0.35
0.88
0.43
0.19
0.13
0.89
0.19
0.14
0.13
sink
cupbound
colfce
MRT
0.36
01.1
0.10
0.81
0.29
0.15
0.30
0.82
0.30
0.12
0.11
<MI [47]
0.90
0.36
0.33
0.19
0.86
0.37
0.22
0.19
0.18
0.86
0.38
0.22
0.19
0.19
CHICO [18]
0.57
0.34
0.22
0.39
0.188
0.84
0.36
0.22
0.19
0..18
0.83
0.36
0.23
0.19
0.18
pgligg [4S]
0.89
0.37
0.23
0.20
0.29
0.86
0.39
0,23
0.20
0.19
0.85
0.39
0.24
0.21
0.20
HisRRp
0.19
0.43
0.15
0.14
motion of each person independently. MRT forecasts the motion of all persons jointly. The source code of all methods is publicly available and we utilize the original hyper-parameteen the input and output dimensions to fit our skeleton representation. The adapted source code of all methods is publicly accessible via our API. We normalize the human poses in a pre-processe rre-processing step as in [46] such that the root joint of the last observed frame is translated to the origin and the hip is axis-aligned with the x-axis. After the forecasting, we apply the inverse transformation convert the forecast motion back into global 3D coordinates. For MRT, we use the normalization proposed in [50], i.e., we shift all kitchens so that the mean pose location is at the origin to prevent drift.
We only adjust
Short- Forecasting:
For short-term forecasting, the methods receive 50 frames (2s) of input motion and forecast 25 frames (1s). As metric we utilize MPJPE. The results in Tab. 4 show that Reery--epeat--IIee
[49] performs best for the first 15 frames, but at frame 25 pgbig [48] performs best for most activities. MRT [50] performed best on the Cupboard sequences at frame 25. We will see in the long-term experiments that History-Repeats--rse Itself is very strong for very short time horizons but not suitable for longer sequences, whereas pgbig [48] is more general approach that performs well also for longer horizons. In general, the lowest errors are observed for Whiteboard, Sink, Cupboard and Coffee since the global position changes less than for the activities Walking and Sitting down.
Long- Long-Term Forecasting: For long-term forecasting, the methods forecast 250 frames (10s) given 50 frames (2s). In general, we suggest for future works to use 250 frames for the observation as well since restricting the observations to 50 frames is not necessary. However, state-of-thee the-art approaches operate on a fixed input window, an important hyper-parameter, that sometimes can only be changed by modifying the architecture. We thus kept the input sequence for the short-term forecasting. As metric, we utilize NDMS with kernel size 8 [46]. The results in Tab. 5 show that all methods produce reasonable motion up I second but the quality deteriorates afterwards. Similar to the short-term forecasting, Reppars--epeass---ee Itself [49] performs best for the shorter time horizons while pgbig [48] performs best for the longer time horizons. Interestingly, MRT [50], which is the only multi-person method, performs worse than the other methods. The difference might be the normalization used in MRT, which is based the mean pose and not the last observed pose. In general, MRT generates good motions for dynamic activities, i.e., walking, sitting down, or standing up, but for motions, which are more subtle, the model often freezes. This suggests that the joint generation of realistic motion of more than 3 persons is challenging. We also find that MRT struggles to keep sensible distances between persons when a scene gets more crowded. For a high number of persons per scene, i.e. more than 8, we found multiple instances of a person walking into another person. This does not occur in scenes with fewer persons. This indicates that MRT is able to utilize information about the distances and relative locations between people, but its performance of doing so deteriorates with more people. Overall, the results show that modeling the interactions of multiple persons is not well handled by the current thte-a.- Furthermore, none of the current approaches is able to incorporate scene context. Humans in Kitchens thus opens new research directions to study multi-person human motion forecasting with scene context.

---

4.1 Discussion
We presented Humans in Kitchen, a large-scale dataset for multi-person person human motion forecasting, consisting of 7.3h hours of recordings of persons interacting in real kitchen environments. The dataset includes over 4M annotated 3D human poses of 90 individuals with unique IDs, 82 annotated activities and 156 annotated objects. It is the first dataset that combines multiple persons, activities, and scene contexts in real setting. We utilize well-known open source formats and the dataset can be easily used with our provided API. We further provided two benchmarks for short and long-term human motion forecasting. We belief that Humans in Kitchens will be a valuable source to advance and evaluate approaches for multi-person human motion forecasting with scene context.
Limitations and Future Directions: While Humans in Kitchens aims to record behaviour as natural as possible, persons might still behave differently from real-life when they know that they recorded. Due to the more natural behaviour, the majority of activities is standing and sitting as shown in Tab. 3. Such unbalanced data is realistic and it can be addressed by evaluating per activity to avoid bias towards these two activities. Another limitation the coarse scene geometry annotation. We decided to prefer coarse but dynamic geometry compared to static geometry since in particular chairs are moved within a 2 hours recording. It might be possible to fit 3D shapes to the objects in the future, but this will not be straightforward since the objects rotate and the orientation of the kettle, for example, is difficult to annotate. Furthermore, we only fit the SMPL model and do not use more detailed body models that include hand pose or more details of the face due to the large degree of occlusion as well as the limited resolution caused by the distance to the cameras. The dataset can thus not be used for modeling fine-grate aneeggriied hand-object interactions. We have also observed that state-of-the offtth--rt approaches struggle to model the motion and interaction of multiple persons. Another open research question is the modeling of scene context for forecasting, in particular as the size of objects varies from 30cm (kettle) to 10m (walls) in width.
Negative Societal Impact:
The purpose of this dataset is to aid research for forecasting approaches for socially aware robots. It needs to be considered that forecasting models can also be misused for monitoring civilians. To address this issue, focused only on kitchen related activities. In case of misuse, we reserve our right to withdraw the permission to use the dataset at any point.
Acknowledgments and Disclosure of Funding
The work has been funded by the Deutsche Forschungsgemeinschattt (DFG, German Research Foundation) 1927/5-2 (FOR 2535 Anticipating Human Behavior) and the ERC Consolidator Grant FORHUE (101044724). Oh-Hun Kwon has been supported by the Deutsche Forschungsgemeinschaatt (DFG, German Research Foundation) under Germany's ' Excellence Strategy EXC 2070 -390732324. The development of this publication was supported by the Ministry of Economic Affairs, Industry, Climate Action and Energy of the State of North Rhine- Weste-Westppalia as part of the flagship project ZERTIFIZIERTE KI.
References
[1] Ronja Möller, Antonino Furnari, Sebastiano Battiato, Aki Härmä and Giovanni Maria Farinella. A survey on human-aware robot navigation. Robotics and Autonomous Systems, 2021.
[2] Timur Bagautdinov, Alexandre Alahi, François Fleuret, Pascal Fua, and Silvio Savarese. So- cial scene understanding: End-to-end multi-person action localization and collective activity recognition. In Conference on Computer Vision and Pattern Recognition, 2017.
[3] Lilli Bruckschen, Sabrina Amft, Julian Tanke, Juergen Gall, and Maren Bennewitz. Detection of generic human-object interactions in video streams. In Social Robotics, 2019.
[4] Julia Kantorovitch, Janne Väre, Vesa Pehkonen, Arto Laikari, and Heikki Seppälä. An assistive household fobot-doing more than just cleaning. Journal of Assistive Technologies, 2014.
[5] Brendan Tran Morris and Mohan Manubhai Trivedi. survey of rision-based trajectory learning and analysis for surveillance. Transactions on Circuits and Systems for Video Technology, 2008.
10

---

[6] Sangmin Oh, Anthony Hoogs, Amitha Perera, Naresh Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit Mukherjee, JK Aggarwal, Hyungtae Lee, Larry Davis, et al. large-scale benchmark dataset for event recognition in surveillance video. In Conference Computer Vision and Pattern Recognition, 2011.
[7] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Conference on Computer Vision and Pattern Recognition, 2016.
[8] Lukasz Kidziński, Bryan Yang, Jennifer L. Hicks, Apoorva Rajagopal, Scott L Delp, and Michael Schwartz. Deep neural networks enable quantitative movement analysis using single-camera videos. Nature communications, 2020.
[9] Caroline J Charpentier and John P O' O'Doherty. The application of computational models to social neuroscience: promises and pitfalls. Social neuroscience, 2018.
[10] Cigdem Beyan, Vittorio Murino, Gentiane Venture, and Agnieszka Wykowska. Computational Approaches for Human-Human and Human-Robot Social Interactions. Frontiers in Robolics and AI, 2020.
[11] Leyla Isik, Anna Mynick, Dimitrios Pantazis, and Nancy Kanwisher. The speed of human social interaction perception. Neurolmage, 2020.
[12] Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic Studio: A Massively Multiview System for Social Motion Capture. In International Conference on Computer Vision, 2015.
[13] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. Transactions Graphics, 2015.
[14] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. International Conference on Computer Vision, 2021.
[15] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In International Conference on Computer Vision, 2019.
[16] Bharat Lal Bhatnagar, Xianghui Xie, Ilya A Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Behave: Dataset and method for tracking human object interactions. In Conference on Computer Vision and Pattern Recognition, 2022.
[17] Nan Jiang, Tengyu Liu, Zhexuan Cao, Jieming Cui, Yixin Chen, He Wang, Yixin Zhu, and Siyuan Huang. Chairs: Towards full-body articulated human-object interaction. arXiv preprint 2. 2022.
[18) Alessio Sampieri, Guido Amely, Andrea Avogaro, Federico Cunico, Geri Skenderi, Francesco Setti, Marco Cristani, and Fabio Galasso. Pose forecasting in industrial human-robot collabora- tion. European Conference on Computer Vision, 2022.
[19] Yang Zheng. Yanchao Yang, Kaichun Mo, Jiaman Li. Tao Yu, Yebin Liu, Karen Liu, and Leonidas J Guibas. Gimo: Gaze-informed human motion prediction in context, In European Conference on Computer Vision, 2022.
[20] Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long- term human motion prediction with scene context. In European Conference on Computer Vision, 2020.
[21] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3 6.6: Large scale datasets and predictive methods for 3d human sensing in natural environments. Pattern Analysis and Machine Intelligence, 2014.
[22] Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Human- ise: Language-conditioned human motion generation 3d scenes. In Advances in Neural Information Processing Systems, 2022.

---

[23] Philipp Kratzer, Simon Bihlmaier, Niteesh Balachandra Midlagajni, Rohit Prakash, Mare Toussaint, and Jim Mainprice. Mogaze: A dataset of full-body motions that includes workspace geometry and eye-gaze. Robotics and Automation Letters, 2020.
[24] Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J Black. Resolving 3d human pose ambiguities with 3d scene constraints. In International Conference on Computer Vision, 2019.
[25] Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black. Stochastic scene-aware motion prediction. In International Conference on Computer Vision, 2021.
[26] Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and moving camera. In European Conference on Computer Vision, 2018.
[27] Mihai Fieraru, Mihai Zanfir, Elisabeta Oneata, Alin-lonut Popa, Vlad Olaru, and Cristian Sminchisescu. Three- dicenssimnnion reconstruction of human interactions. In Conference on Computer Vision and Pattern Recognition, 2020.
[28] CMU. Carnegie-Mellon Mocap Database.
[29] Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan. Lin Gui, Sean Banerjee, Timo- thy Scott Godisart, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social interaction capture. Transac- tions on Pattern Analysis and Machine Intelligence, 2017.
[30] Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Mare Pollefeys, Federica Bogo, and Siyu Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. In European conference on computer vision, 2022.
[31] Wen Guo, Xiaoyu Bie, Xavier Alameda-Pineda, and Francese doreno-Noguer. extreme motion prediction. In Conference on Computer Vision and Pattern Recognition, 2022.
[32] Hanbyul Joo, Tomas Simon, Mina Cikara, and Yaser Sheikh. Towards social artificial intelli- gence: Nonverbal social signal prediction in a triadic interaction. In Conference on Computer Vision and Pattern Recognition, 2019.
[33] Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, and Christian Theobalt. Single-shot multi-person 3d pose estimation from monocular rgb. In International Conference 3D Vision, 2018.
[34] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C Kot. Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding. Transactions on Puttern Analysis and Machine Intelligence, 2019.
[35] Chun-Har Hao P. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J. Black. Capturing and inferring dense full-body human-scene contact. In Computer Vision and Puttern Recognition, 2022.
[36] Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J Black. Populating 3d scenes by learning human-scene interaction. In Conference Computer Vision and Puttern Recognition, 2021.
[37] Zhitao Yang, Zhongang Cai, Haiyi Mei, Shuai Liu, Zhaoxi Chen, Weiye Xiao, Yukun Wei, Zhongfei Qing, Chen Wei, Bo Dai, Wayne Wu, Chen Qian, Dahua Lin, Ziwei Liu, and Lei Yang. Synbody: Synthetic dataset with layered human models for 3d human perception and modeling. arXiv preprint arXiv: 17668, 2023.
[38] Angela Dai, Matthias NieBner, Michael Zollöfer, Shahram Izadi, and Christian Theobalt. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface re- integration. Transactions Graphics, 2017.
12

---

[39] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart. Ahmed A. A. Osman, Dimitrios Tzionas, and Michael Black. Expressive body capture: 3d hands, face, and body from single image. In Conference on Computer Vision and Pattern Recognition, 2019.
[40] Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Conference on Computer Vision and Pattern Recognition. 2016.
[41] Julian Tanke, Oh-Hun Kwon, Patrick Stotko, Radu Alexandru Rosu, Michael Weinmann, Hassan Errami, Sven Behnke, Maren Bennewitz, Reinhard Klein, Andreas Weber, et al. Bonn activity maps:
Dataset description. arXiv preprint arXiv: 2019.
[42] Julian Tanke and Juergen Gall. Iterative greedy matching for 3d human pose tracking from multiple views. In German Conference on Puttern Recognition, 2019.
[43] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime Multi-Person 21 Pose Esti- mation using Part Affinity Fields. In Conference on Computer Vision and Pattern Recognition, 2017.
[44] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. International Conference on Learning Representations, 2023.
[45] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Conference on Computer Vision and Pattern Recognition, 2017.
[46] Julian Tanke, Chintan Zaveri, and Juergen Gall. intention-based long-term human motion anticipation. In 2021 International Conference on 3D Vision, 2021.
[47] Wen Guo, Yuming Du, Xi Shen, Vincent Lepetit, Xavier Alameda- Pineda,, and Francesc Moreno- Noguer. Back to mlp: A simple baseline for human motion prediction. In Winter Conference on Applications of Computer Vision, 2023.
[48] Tiezheng Ma, Yongwei Nic. Chengjiang Long, Qing Zhang, and Guiqing Li. Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In Conference on Computer Vision and Pattern Recognition, 2022.
[49] Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion prediction via motion attention. In European Conference on Computer Vision, 2020.
[50] Jiashun Wang. Huazhe Xu, Medhini Narasimhan, and Xiaolong Wang. Multi-person 3d motion prediction with multi multi-range transformers. Advances Neural Information Processing Systems, 2021.
13


Reviews:
[{"review_id": "0qqooejF2U", "reviewer": "Reviewer_zmcC", "summary": "", "questions": "", "limitations": "The authors have discussed this paper's limitations and the societal impact.\n", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper proposes a large-scale multi-person motion dataset containing more than 4M annotated human bodies and frame-wise activity for each person in 4 kitchen scenes. The recorded data involves up to 16 individuals at the same time within a single scene.\n\n2. The authors provide a comprehensive description of the dataset's acquisition, annotation, and statistical analysis.\n\n3. They thoroughly evaluate several state-of-the-art methods for the human motion forecasting task on the proposed dataset, comprising both short-term and long-term forecasting settings.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "TjtYhTR37H", "reviewer": "Reviewer_khgs", "summary": "", "questions": "", "limitations": "This work discusses some limitations of the main paper. However, I think there are additional limitations that should be discussed, as shown in the above part (about the SMPLX annotations, multi-person motion forecasting models, pre-trained forecasting models to generalize well across datasets).", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper is easy to read and provides comprehensive quantitative and qualitative information about the dataset.\n2. This dataset addresses an interesting and important direction for 3D multi-person scenes with interactions and occlusions. It provides a large-scale dataset with 3D human pose annotations and 3D boxes for objects. These useful data will benefit 3D human-object interaction and behavior understanding.\n3. This work provides comprehensive visualization to understand and validate the data.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "BHhXGo3Cja", "reviewer": "Reviewer_Kghq", "summary": "", "questions": "", "limitations": "1. There are only a few hours of video in the dataset. These could have been recorded in a day and it is not really \"large-scale\". Compared to prior works it is large scale, and the quality and density of annotations is impressive (which is probably more important), but it's still only 7.33hr. \n2. Pose estimation for many hours of people standing is not that interesting. Most existing 3D keypoint/motion datasets involve physical activities with lots of variation in pose. It's unclear that this dataset is sufficiently diverse to support modeling the motion of multiple people engaged in more interesting/varied activities.\n3. As is, it feels like some additional analyses of the quality of their data are missing. Specifically, more analysis of how the new dataset differs from synthetic data and how existing methods fail on Humans in Kitchens would be valuable.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Humans in Kitchens is a valuable new dataset for a number of reasons:\n\n1. Existing datasets (outside of EgoBody) don't include multiple people and their 3D poses in larger scene coordinates. \n2. Recent works like MDM show the promise of diffusion at learning generative models of human motion. However, those models generate motion in a vacuum, where the motion is only for a single person in an unknown 3D scene. Humans in Kitchens addresses an important *data gap* (multiple people in a scene) that existed for motion generation. Using Humans in Kitchens, one could now produce models that condition on multiple people when synthesizing a forecasted 3D motion.\n3. At 7.33hr, this is quite a large dataset. I don't think I would consider it \"large-scale\" but the dense annotations definitely make it invaluable for the task of human motion forecasting you've targeted.\n4. This dataset is relevant to the broader research community as methods in computer vision, reinforcement learning, and robotics can benefit from an understanding of multiple people moving in 3D. More specifically, understanding people as agents has been a compelling direction for these fields and although progress has been made in what can be seen, i.e., computer vision, progress is still lagging in terms of what embodied agents can accomplish and how they can move conditioned on their vision. Having this multi-person 3D keypoint dataset is valuable for furthering this effort, expanding opportunities for combining 3D keypoint motion synthesis with vision.\n5. We had 3D keypoint datasets already in AMASS, and we have lots of 2D keypoint datasets with multiple people.. but having this Humans In Kitchens dataset fills in a missing blank while being real world enough to probably be useful.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "kDx7fHVIbr", "reviewer": "Reviewer_1cBr", "summary": "", "questions": "", "limitations": "Apart from the aforementioned opportunities for improvement, another major limitation of this work is its insufficient generalization ability.\nThe authors themselves acknowledge the complexities of social and object-person interactions in various outdoor and indoor environments.\nHowever, the dataset provided and the framework developed in this paper are limited to humans in kitchens, which severely restricts its applicability to other environmental scenarios. It is a significant limitation.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper is well written, and dataset statistics and data collection details are clearly mentioned.\n2. Multiple SOTA motion prediction methods for single and multiple individuals are evaluated. The authors report performance under two prediction settings, providing useful benchmarks for further work.\n3. Addressing the existing gap in current motion datasets, the authors aim to capture more motions of interactive subjects and human-scene interactions. The incorporation of this large-scale dataset is envisioned to contribute to advancing research in this domain.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "OozoAGvg0f", "reviewer": "Reviewer_bd4f", "summary": "", "questions": "", "limitations": "The authors discussed some of the limitations which I agree with.\n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The provided data is very valuable, especially the per-frame activity label attached to 3D human instances and the dynamic scene geometry.\nIt would considerably facilitate the research on human-human/scene interactions.\n\nWith multiple off-the-shelf tools and considerable human labor, the annotation pipeline manages to provide fine-grained annotations.\n\nModelling scene geometry as boxes is a reasonable compromise.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents a dataset with multi-person human motions in scenes. The dataset is novel and interesting in its setting as it enables future research on human motion forecasting in scenes where multiple persons could be present. AC and all reviewers have reached a clear consensus regarding this point. The authors are encouraged to address the following weakness points in the final camera-ready version, however: 1) tone down some claims and rewrite the related paragraphs to make things clearer; 2) discuss a plan to add more videos in novel scenes or humans doing novel tasks; 3) present more thorough analysis and statistics for the dataset and the benchmarking experiments. AC and reviewers all agree on the acceptance recommendation of this paper.
Decision: Accept (Poster)