SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The dataset and API are free to download<sup>6</sup> and can be used for non-commercial purposes. The API is under MIT-License while the dataset utilizes a custom license. All subjects signed forms consenting that recordings of them or derived of them can be used for non-commercial research purposes. The recording has been approved by the ethical review committee of the University of Bonn. In contrast to the raw data [41], Humans in Kitchens does not contain personally identifiable information. The raw data is also accessible<sup>7</sup>, but requires to sign a license agreement and is subject to export regulations. Note that the raw data is not required for using Humans in Kitchens. The data does not contain any offensive content.
We evaluate various state-of-the-art methods on the human motion forecasting task of our dataset. More precisely, the goal is to learn a function  $f$  that takes as input a human pose sequence  $\mathbf{X}^{1:t} = (\mathbf{x}_1^{1:t},\mathbf{x}_2^{1:t},\dots ,\mathbf{x}_n^{1:t})$  of  $n$  persons, where  $\mathbf{x}_i^{1:t}\in \mathbb{R}^{t\times (29\times 3)}$  represents a 3D motion sequence of  $t$  frames for person  $i$ , and forecasts the future motion for all  $n$  persons in global coordinates:
$$
\hat {\mathbf {X}} ^ {t + 1: T} = f \left(\mathbf {X} ^ {1: t}\right), \quad \hat {\mathbf {X}} ^ {t + 1: T} \in \mathbb {R} ^ {(T - t) \times n \times (2 9 \times 3)}. \tag {1}
$$
As our dataset contains natural behavior over very long time, we select 2 interesting motion activities, namely Walking and Sitting down, and 4 interesting interactions with the objects Whiteboard, Sink, Cupboard, and Coffee Machine for evaluation. Note that most of these actions involve social interactions since the persons sit in groups and discuss at the whiteboard. For training, we use the kitchens A, B and C and we evaluate on D, which is the largest among the four kitchens. For the human-object interactions, we select the last observed frame  $t$  as the first frame of the annotated action, while for the motion activities we select frame  $t - 10$  such that a few frames of the motion are already observed. Overall, we sample all occurrences of the given activity in the test set<sup>8</sup>. We only evaluate the forecast motion of the person that performs the activity and not for any other person in the scene. This allows to report accuracy per activity, but it still requires to model the context of the other persons. We consider two distinct protocols: short-term and long-term motion forecasting. In the short-term protocol, we employ the widely used Mean Per Joint Positional Error (MPJPE) metric [45] to measure the positional disparity between the predicted motion and the ground truth. For the long-term protocol, we use the Normalized Directional Motion Similarity Score (NDMS) [46], which effectively assesses the quality of motion sequences of any given length.
**Baseline methods:** We evaluate 4 recent single-person human motion forecasting methods, namely siMLPe [47], CHICO [18], pgbig [48] and History-Repeats-Itself [49], and the multi-person forecasting method Multi-Range Transformers (MRT) [50]. While the single-person approaches forecast the
$^{6}$ https://github.com/jutanke/hik
<sup>7</sup>https://github.com/bonn-activity-maps/bonnActivity_maps
8 More details are provided in the supplementary material
Table 4: MPJPE  $\downarrow$  in dm. Methods denoted with * forecast all persons at the same time.
frame	5	15	25	5	15	25	5	15	25	5	15	25	5	15	25	5	15	25
walking	sitting down	whiteboard	sink	cupboard	coffee
MRT* [50]	0.40	0.91	1.40	0.38	0.97	1.42	0.24	0.56	0.80	0.29	0.80	1.16	0.28	0.61	0.89	0.26	0.69	1.19
siMLPe [47]	0.39	0.97	1.59	0.37	0.98	1.40	0.24	0.53	0.72	0.17	0.43	0.59	0.23	0.64	0.98	0.22	0.55	0.88
CHICO [18]	0.37	0.88	1.42	0.36	0.96	1.36	0.27	0.55	0.69	0.19	0.46	0.61	0.23	0.65	1.05	0.22	0.55	0.86
pgbig [48]	0.34	0.85	1.40	0.34	0.87	1.22	0.23	0.52	0.66	0.16	0.42	0.58	0.20	0.59	0.93	0.20	0.54	0.86
HistRep [49]	0.30	0.83	1.57	0.30	0.84	1.23	0.19	0.48	0.68	0.12	0.40	0.57	0.16	0.57	0.99	0.17	0.50	0.95
Table 5: NDMS  $\uparrow$ . Methods denoted with * forecast all persons at the same time.
frame	1	25	100	200	250	1	25	100	200	250	1	25	100	200	250
walking	sitting down	whiteboard
MRT* [50]	0.81	0.34	0.18	0.13	0.12	0.81	0.28	0.14	0.11	0.10	0.80	0.26	0.14	0.11	0.10
siMLPe [47]	0.87	0.41	0.24	0.20	0.19	0.87	0.37	0.22	0.19	0.18	0.88	0.35	0.21	0.18	0.17
CHICO [18]	0.84	0.40	0.25	0.20	0.19	0.85	0.36	0.22	0.19	0.18	0.86	0.35	0.22	0.18	0.17
pgbig [48]	0.86	0.43	0.25	0.21	0.20	0.87	0.38	0.23	0.20	0.19	0.87	0.37	0.23	0.20	0.19
HistRep [49]	0.87	0.46	0.21	0.15	0.14	0.88	0.43	0.19	0.14	0.13	0.89	0.43	0.19	0.14	0.13
sink	cupboard	coffee
MRT* [50]	0.81	0.26	0.13	0.10	0.10	0.81	0.29	0.15	0.11	0.10	0.82	0.30	0.16	0.12	0.11
siMLPe [47]	0.90	0.36	0.22	0.19	0.18	0.86	0.37	0.22	0.19	0.18	0.86	0.38	0.22	0.19	0.19
CHICO [18]	0.87	0.34	0.22	0.19	0.18	0.84	0.36	0.22	0.19	0.18	0.83	0.36	0.23	0.19	0.18
pgbig [48]	0.89	0.37	0.23	0.20	0.20	0.86	0.39	0.23	0.20	0.19	0.85	0.39	0.24	0.21	0.20
HistRep [49]	0.90	0.43	0.19	0.14	0.13	0.87	0.43	0.19	0.14	0.13	0.86	0.44	0.20	0.15	0.14
motion of each person independently, MRT forecasts the motion of all persons jointly. The source code of all methods is publicly available and we utilize the original hyper-parameters. We only adjust the input and output dimensions to fit our skeleton representation. The adapted source code of all methods is publicly accessible via our API. We normalize the human poses in a pre-processing step as in [46] such that the root joint of the last observed frame  $t$  is translated to the origin and the hip is axis-aligned with the x-axis. After the forecasting, we apply the inverse transformation to convert the forecast motion back into global 3D coordinates. For MRT, we use the normalization proposed in [50], i.e., we shift all kitchens so that the mean pose location is at the origin to prevent drift.
Short-Term Forecasting: For short-term forecasting, the methods receive 50 frames (2s) of input motion and forecast 25 frames (1s). As metric we utilize MPJPE. The results in Tab. 4 show that History-Repeats-Itself [49] performs best for the first 15 frames, but at frame 25 pgbig [48] performs best for most activities. MRT [50] performed best on the Cupboard sequences at frame 25. We will see in the long-term experiments that History-Repeats-Itself is very strong for very short time horizons but not suitable for longer sequences, whereas pgbig [48] is a more general approach that performs well also for longer horizons. In general, the lowest errors are observed for Whiteboard, Sink, Cupboard and Coffee since the global position changes less than for the activities Walking and Sitting down.
Long-Term Forecasting: For long-term forecasting, the methods forecast 250 frames (10s) given 50 frames (2s). In general, we suggest for future works to use 250 frames for the observation as well since restricting the observations to 50 frames is not necessary. However, state-of-the-art approaches operate on a fixed input window, an important hyper-parameter, that sometimes can only be changed by modifying the architecture. We thus kept the input sequence as for the short-term forecasting. As metric, we utilize NDMS with kernel size 8 [46]. The results in Tab. 5 show that all methods produce a reasonable motion up to 1 second but the quality deteriorates afterwards. Similar to the short-term forecasting, History-Repeats-Itself [49] performs best for the shorter time horizons while pgbig [48] performs best for the longer time horizons. Interestingly, MRT [50], which is the only multi-person method, performs worse than the other methods. The difference might be the normalization used in MRT, which is based on the mean pose and not the last observed pose. In general, MRT generates good motions for dynamic activities, i.e., walking, sitting down, or standing up, but for motions, which are more subtle, the model often freezes. This suggests that the joint generation of realistic motion of more than 3 persons is challenging. We also find that MRT struggles to keep sensible distances between persons when a scene gets more crowded. For a high number of persons per scene, i.e., more than 8, we found multiple instances of a person walking into another person. This does not occur in scenes with fewer persons. This indicates that MRT is able to utilize information about the distances and relative locations between people, but its performance of doing so deteriorates with more people. Overall, the results show that modeling the interactions of multiple persons is not well handled by the current state-of-the-art. Furthermore, none of the current approaches is able to incorporate scene context. Humans in Kitchens thus opens new research directions to study multi-person human motion forecasting with scene context.
4.1 Discussion
We presented Humans in Kitchen, a large-scale dataset for multi-person human motion forecasting, consisting of 7.3h hours of recordings of persons interacting in real kitchen environments. The dataset includes over 4M annotated 3D human poses of 90 individuals with unique IDs, 82 annotated activities and 156 annotated objects. It is the first dataset that combines multiple persons, activities, and scene contexts in a real setting. We utilize well-known open source formats and the dataset can be easily used with our provided API. We further provided two benchmarks for short and long-term human motion forecasting. We believe that Humans in Kitchens will be a valuable source to advance and evaluate approaches for multi-person human motion forecasting with scene context.
Limitations and Future Directions: While Humans in Kitchens aims to record behaviour as natural as possible, persons might still behave differently from real-life when they know that they are recorded. Due to the more natural behaviour, the majority of activities is standing and sitting as shown in Tab. 3. Such unbalanced data is realistic and it can be addressed by evaluating per activity to avoid a bias towards these two activities. Another limitation is the coarse scene geometry annotation. We decided to prefer a coarse but dynamic geometry compared to a static geometry since in particular chairs are moved within a 2 hours recording. It might be possible to fit 3D shapes to the objects in the future, but this will not be straightforward since the objects rotate and the orientation of the kettle, for example, is difficult to annotate. Furthermore, we only fit the SMPL model and do not use more detailed body models that include hand pose or more details of the face due to the large degree of occlusion as well as the limited resolution caused by the distance to the cameras. The dataset can thus not be used for modeling fine-grained hand-object interactions. We have also observed that state-of-the-art approaches struggle to model the motion and interaction of multiple persons. Another open research question is the modeling of scene context for forecasting, in particular as the size of objects varies from  $30\mathrm{cm}$  (kettle) to  $10\mathrm{m}$  (walls) in width.
Negative Societal Impact: The purpose of this dataset is to aid research for forecasting approaches for socially aware robots. It needs to be considered that forecasting models can also be misused for monitoring civilians. To address this issue, we focused only on kitchen related activities. In case of misuse, we reserve our right to withdraw the permission to use the dataset at any point.
The work has been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) 1927/5-2 (FOR 2535 Anticipating Human Behavior) and the ERC Consolidator Grant FORHUE (101044724). Oh-Hun Kwon has been supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC 2070 -390732324. The development of this publication was supported by the Ministry of Economic Affairs, Industry, Climate Action and Energy of the State of North Rhine-Westphalia as part of the flagship project ZERTIFIZIERTE KI.
[1] Ronja Möller, Antonino Furnari, Sebastiano Battiato, Aki Härma, and Giovanni Maria Farinella. A survey on human-aware robot navigation. Robotics and Autonomous Systems, 2021.
[2] Timur Bagautdinov, Alexandre Alahi, François Fleuret, Pascal Fua, and Silvio Savarese. Social scene understanding: End-to-end multi-person action localization and collective activity recognition. In Conference on Computer Vision and Pattern Recognition, 2017.
[3] Lilli Bruckschen, Sabrina Amft, Julian Tanke, Juergen Gall, and Maren Bennewitz. Detection of generic human-object interactions in video streams. In Social Robotics, 2019.
[4] Julia Kantorovitch, Janne Väre, Vesa Pehkonen, Arto Laikari, and Heikki Seppälä. An assistive household robot-doing more than just cleaning. Journal of Assistive Technologies, 2014.
[5] Brendan Tran Morris and Mohan Manubhai Trivedi. A survey of vision-based trajectory learning and analysis for surveillance. Transactions on Circuits and Systems for Video Technology, 2008.
[39] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. In Conference on Computer Vision and Pattern Recognition, 2019.
[40] Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Conference on Computer Vision and Pattern Recognition, 2016.
[41] Julian Tanke, Oh-Hun Kwon, Patrick Stotko, Radu Alexandru Rosu, Michael Weinmann, Hassan Errami, Sven Behnke, Maren Bennewitz, Reinhard Klein, Andreas Weber, et al. Bonn activity maps: Dataset description. arXiv preprint arXiv:1912.06354, 2019.
[42] Julian Tanke and Juergen Gall. Iterative greedy matching for 3d human pose tracking from multiple views. In German Conference on Pattern Recognition, 2019.
[43] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. In Conference on Computer Vision and Pattern Recognition, 2017.
[44] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. International Conference on Learning Representations, 2023.
[45] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Conference on Computer Vision and Pattern Recognition, 2017.
[46] Julian Tanke, Chintan Zaveri, and Juergen Gall. Intention-based long-term human motion anticipation. In 2021 International Conference on 3D Vision, 2021.
[47] Wen Guo, Yuming Du, Xi Shen, Vincent Lepetit, Xavier Alameda-Pineda, and Francesc Moreno-Noguer. Back to mlp: A simple baseline for human motion prediction. In Winter Conference on Applications of Computer Vision, 2023.
[48] Tiezheng Ma, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In Conference on Computer Vision and Pattern Recognition, 2022.
[49] Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion prediction via motion attention. In European Conference on Computer Vision, 2020.
[50] Jiashun Wang, Huazhe Xu, Medhini Narasimhan, and Xiaolong Wang. Multi-person 3d motion prediction with multi-range transformers. Advances in Neural Information Processing Systems, 2021.