Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors proposed a multi-modal 3D human pose estimation dataset, targeting rehabilitation exercises. The collected the dataset using 20 subjects. They combined mmWave, RGB-D, and IMUs as sensing modalities, which is different from most of the prior works that are largely focusing on RGB, Lidar and the likes. 

They documented the dataset generation details, including methods of data collection and annotation. The authors provided benchmark of several models using their dataset for all of their modalities. They ended the paper with discussion on ethical aspects, conclusion and future work.




strengths: The authors managed to identify the gap from prior works, and thus proposed their dataset for 3D human pose estimation. This dataset offers some beneficial properties i.e. low cost, privacy, and low computational power. 

Their multi-modality also allows for more creative adaptation of application using this dataset. The dataset focuses on healthcare rehabilitation, making it having high relevance to such usage versus some other generic datasets.  

Apart from comparing their dataset against several closely related prior works, the authors also conducted some extensive benchmark using their dataset.  
weaknesses: In Line 194 - 196, the authors mentioned about the detection of 2D keypoints using HRNet, and they confirmed that these detected 2D keypoints are reasonably accurate. However, there is no details about their approach of confirming the accuracy of these 2D keypoints. These details could be important as they are directly affecting the generated dataset. 
correctness: The dataset is documented and organized in a standard manner.

The benchmarks were evaluated using standard metric.
clarity: The sections are well-organized, and easy for readership. Each point is clearly written and ideas were presented with clarity. Arguments were properly elaborated, with supportive descriptions. 
relation to prior work: Discussion of prior works was given, and the authors listed some major relevant datasets for comparison. The authors managed to point out their work differences as compared to the prior works.
additional feedback: Since the authors stated their benchmark results and some claims on different settings / modality that showed different performances, it will be important that the code implementation / trained models / configurations are given in the landing page / Github site of the dataset for reproduction (for the purpose of evaluation by the reviewers or other researchers).
review_title: A Multi-modal 3D Human Pose Estimation Dataset with Detailed Documentation and Extensive Benchmarking

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper describes the mRI dataset, a multi-modal 3D human pose estimation dataset with mmWave, RGB-D, and Inertial Sensors. mRI consists of over 5 million frames from 20 subjects performing rehabilitation exercises. The authors give benchmarks for the dataset in two categories - 3D human pose estimation and action detection. 3D Human Pose Estimation has two evaluation protocols - direct 3D pose evaluation and cross-subject generalization.

The main contribution of this dataset is that it supports multiple sensing modalities. Besides, mRI focuses on healthcare and only requires low-power and low-cost devices.

strengths: + Paper is well written. The dataset statistics and data collection details are clearly mentioned.
+ It is the first dataset with the most comprehensive set of sensing modalities, including RGB, depth, IMU, and mmWave.xnww 
weaknesses: - This work proposes a multi-modal dataset but only evaluates the standard methods for each modality separately. As shown in Table 3, while using only the IMU obtains accurate results (40.9 MPJPE and 28.4 PA-MPJPE), it is not clear whether using multiple modalities is beneficial and necessary. 

- The 3D keypoints are labeled with the RGB-based 2D pose estimation method with triangulation, which is not robust and vulnerable to occlusion. Therefore, the ground-truth 3D poses are not accurate. To ensure accurate 2D pose estimation, the RGB image should be captured in a controlled environment, while there are already lots of indoor datasets that provide 3D pose annotations.

- In table 3, the methods using mmWave, RGB, and IMUs are trained in different settings. Thus the direct comparison is not fair.


correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: I suggest the authors report the results of the RGB-based 3D pose estimator that is fine-tuned in mRI.
review_title: Official Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a multi-modal dataset of mmWave, RGB-D, and IMU readings during rehabilitation-focused movements in healthy subjects.  It spans 20 subjects and includes 12 labels (10 motions/exercises and 2 free-form periods).  Joint angles are determined via an optimization procedure, and activity labels are provided.  The paper also presents results about extracting joint angles from each sensor subset, and about classifying activities based on joint angles.
strengths: It is valuable to have multi-modal datasets, and facilitating at-home rehabilitation with personalized feedback is laudable.  The dataset contains a good mix of sensors that may facilitate cross-modal analysis.  The benchmarks address the common tasks of joint reconstruction and activity recognition.
weaknesses: A few questions/comments that came to mind are summarized below:

- The paper described the dataset as comprising over 5 million frames, but this does not seem like a particularly useful size metric; I think duration would be more meaningful.  I'm not actually sure if the paper mentioned metrics such as duration (per subject, per activity, etc.), repetitions of activities, etc.  These should be added to better describe the dataset.  The number of synced frames was reported, which comes closer to measuring duration, but it is still unclear.

- Table 1 says that some papers report synchronized frames and others total frames, but it's not clear which is which in the table (so that column may be an unfair comparison).

- It seems like the optimization method is used to estimate body poses from the data, and then those are used as ground truth for evaluating the other pipelines that also aim to estimate joint positions?  The underlying assumption thus seems to be that the optimization method is much more accurate than the learning pipelines, even though its accuracy cannot be thoroughly evaluated?  Some discussion about the possible implications of this may be good to add.  (Sorry if I'm missing that there was another source of ground truth for the benchmarks.)

- A bit more discussion about why the two cameras are next to each other may be helpful.  I had originally assumed that they would provide different vantage points to avoid occlusions, but now I suspect they are just to provide stereo data?  This may be redundant with the depth data though, unless the aim is to explore stereo vision on its own (perhaps using depth data as ground truth)?

- It was mentioned that the six IMUs capture richer pose representations than the mmWave radar; some more elaboration on that may be useful, since it's not immediately clear given that the radar is providing a point cloud encompassing the entire person if I am understanding it correctly.

- Some more information about the activity detection approach may be helpful, such as the duration of the segments being classified and whether it can operate in a real-time streaming fashion or assumes segmented examples.  Thoughts on why certain modalities are better than others for certain activities may be useful as well.  A confusion matrix may also be interesting.

- The ethics section should probably be in the main paper portion before the conclusion?  Also, the last sentence saying that the approach "does not leverage any bias in the datasets used for validation" was a bit unclear.  There is undoubtedly some bias in the dataset, such as from the recruitment pool yielding similar subject backgrounds or abilities, so mentioning some possible sources (or at least words of caution) may be helpful.  

- Section 3.1 mentions that the slowest sensor is used as the base timestamp, and then the closest sample from each sensor is found.  Does this imply that all sensors are being resampled to the lowest sampling rate?  I hope all data is provided at their original sampling rates in the dataset as well, rather than that data being discarded?  Downsampling may be useful for some applications, but can be a significant drawback for others.  Update: I see that the GitHub includes raw data, but it would be helpful to mention this in the paper as well (along with more information about the dataset structure and formats).

- Left and right may be swapped in section 3.2 when describing Figure 3?
correctness: The evaluation methods seem reasonable, especially the inclusion of leave-one-subject-out validation.  It is a bit hard to evaluate the dataset construction as it is not yet available and little information about its format is provided.  The data collection procedure sounds reasonable though.
clarity: The paper is generally well written, although there are typos and grammatical adjustments to address.
relation to prior work: The related work section nicely discusses relevant areas including human pose detection, related datasets, and rehabilitation.  The table comparing datasets seems comprehensive and helpful.
additional feedback: Congrats on your experiments!
review_title: mRI: Multi-modal 3D Human Pose Estimation Dataset using mmWave, RGB-D, and Inertial Sensors

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces a new multi-modal dataset which can be used for training models to estimate 3D human pose. The dataset contains three modalities namely: IMU sensors, RGB-D, and mm Wave. The captured data are annotated with 2D key points, 3D key points/skeleton and actions (start and end timestamp) performed by the subject. The paper also demonstrates how different modalities perform the tasks of human pose estimation and action detection.
strengths: 1. It is the first dataset to have all three modalities (IMU, RGB-D and mm Wave) that are synchronized. This is an important contribution that could be useful for training models with multi-modalities for pose estimation and action detection. It can also be useful for pretraining action detection/pose estimation models using multi-modal self-supervised learning.

2. The dataset also annotations for action labels, 2D key points and 3D key points. These annotations can be useful for training supervised models. These models can be useful starting points which can be further finetuned to perform action recognition or pose estimation tasks.

3. In comparison to other pose estimation datasets, this dataset has the most number of synchronize frames. This make the dataset suitable for training models for human action understanding.

4. The dataset is relatively well documented.

weaknesses: There are few concerns. If authors can resolve them, it would increase the quality of submission.
1. Quality of the labels: How many annotators annotated the dataset (action labels)? If there were multiple annotators, what was their inter-rater reliability coefficient?

For the key points annotations, how were the model generated key points validated? Were they validated in ad-hoc manner or some proper statistical sampling performed to ensure robust validation?

 What are chances that key point annotations generated by the model are noisy? Is there a way to quantify it (using some kind of sampling and manual inspection)?

2. To ensure reproducibility of the results and benchmarking, it would be advisable to share the train and test splits for each protocol (S1 and S2).

3. How many trials were performed by each subject? Were  subjects free to do any of the 12 movements at any time or was there an experimental protocol followed that instructed the subjects what to do?

4. Currently, the dataset has only young and healthy adults. It would be ideal if the diversity of the data is increased by including olde/elder subjects and possibly subjects with some kind of motor impairment.

5.  To ensure reproducibility of the results, it would be advisable to release the code for training the benchmark models.

6. Since this the first dataset which synchronously captures three modalities, it would be interesting to see how the performance for pose estimation and action detection improves if the model is trained on multiple modalities.
correctness: Yes
clarity: Yes, the paper is well written.
relation to prior work: Yes
additional feedback: None
review_title: A rich multi-modal dataset for pose estimation and action detection

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper collect a multi-modal 3D human pose estimation dataset with mmWave, RGB-D, and 9 Inertial Sensors to enable future research on multiple modalities on home-based health monitoring.
strengths: A new dataset for 3D human pose estimation on home-based health monitoring, contains the data from multiple sensors including mmWave, RGB-D, and 9 Inertial Sensors.
weaknesses: # Poor Evidence/Experiments:
The experiments are too weak so that it does not prove this multi-modal 3D dataset to be useful and needed. 
* The dataset paper does not explore how to fuse data from different sensors to get better results, so that can prove multiple sensors are better than any single one.
* A very important experiment is to prove that this dataset contains all three devices (RGB-D, IMU, mmWave) are better than two devices (RGB-D and IMU); Or Considering privacy, the paper does not prove why this extra mmWave sensor or IMU sensor is useful compared with previous existing datasets (MoVi[11] or MARS[2]) without using RGB data.

# Not Clear Comparison Table 1
If the dataset is already released, it would be much better to count all required numbers for the whole table. 
For example: 
* MoVi[11] has released the dataset, it has more subjects, more sequences, and more action numbers, it potentially has more frames, it would be much better if the author can calculate the number instead of "not listing".

# Not Obvious Visualization
The dataset is for **"3D"** human pose estimation, it should not use only one 2D screenshot for visualization of 3D skeletons, which is quite ambiguous. It should visualize multiple views (bird-view, side-view) of the results or add extra projection on x-z and x-y planes.

The visualization for mmWave is also too bad, does not get how to generate that "random" points for that person from the mmWave radar.
And the visualization of this dataset are not obvious.

# What's the distance between the captured human and the kinect V2?
correctness: Yes. It is a dataset paper. And for those insufficient claims, see in Weaknesses.

clarity: Yes.
relation to prior work: Not that clear. See in Weakness.
additional feedback: Nope.
review_title: Rejection: Weak Experiments and Bad Visualization.

Meta-review: metareview: This paper presents a dataset of synchronized radar, intertial sensors and RGB-D images of healthy young people conducting rehabilitation exercises. The paper also demonstrates how this dataset can be used as a benchmark for human pose estimation and as a catalyst for multi-modal learning. Reviewers appreciated these aspects, while pointing out issues regarding presentation, bias and claims in the analysis. These issues appear well addressed in the revisions. I recommend accepting this paper to the NeurIPS 2022 Dataset and Benchmarks program.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept