Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a framework for recording synchronized multimodal data from wearable and global sensors in the real world to enable a learning pipeline to better understand human everyday behavior and improve in-home robotic assistants.

In addition to making the data publicly available, open-source code and instructions for reproducing the presented framework are available.
strengths: The paper is excellent in that it simultaneously measures wearable sensors in addition to the environment-mounted sensing that has been done in action recognition to date in order to achieve a better understanding of human daily activities and a learning pipeline to improve in-home robotic assistants.

The fact that it targets kitchen activities that include two-handed operation and high-level task planning as tasks is also considered a valid future application.
weaknesses: The name of the dataset is not appropriate. The name of the dataset is limited to kitchen activities, but it is misleading to think that it covers ActionNet and general activities.

The superiority of this dataset should be indicated precisely in the comparison table with the dataset described in the Related Work section. The number of sensors, the number of recorded actions, and the number of samples do not clearly indicate the superiority of this dataset.

As a publisher of the dataset, shouldn't you benchmark the dataset with a basic model and indicate the baseline in the dataset? When the dataset is cited in other papers, it will be difficult to verify the performance, and the validity of the dataset will be reduced.
correctness: The paper is correct for the most part.
clarity: The paper is clear and well written.
relation to prior work: Isn't Action Genome more appropriate than Visual Genome [11] as a prior literature on action recognition?

Action Genome: Actions as Composition of Spatio-temporal Scene Graphs, CVPR2020
https://www.actiongenome.org/
additional feedback: In the case that both hands are involved, it may be more effective to take not only the Tactile sensor but also the hand-pose in order to divert the approach to the object. However, since simultaneous measurement seems to be difficult, it is not included in the Weakness.
If you consider this in the future, the effectiveness of this dataset will be enhanced.
review_title: Review of ActionNet: A Multimodal Dataset for Human Activities Using Wearable Sensors in a Kitchen Environment

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors present a framework to record multimodal data from multiple wearable and visual sensors. Using this framework they collected human activity dataset. The data samples contain a sequence of activities with multimodal sensor data.
strengths: - The authors developed a framework to record multimodal sensor data synchronously.
- The author collected human activity from multiple diverse wearable and sensor data.
weaknesses: - As the authors claim that one of the contributions of this work is to develop a synchronized multimodal data recording system, I would expect how it is different from the existing multimodal data collection system, such as PSI(Platform for Situated Intelligence: https://github.com/microsoft/psi). What is the novelty of this data collection framework compared to the existing data collection frameworks?
- Additionally, I missed the details of their proposed synchronized data collection framework. More specifically, I would like to know more about the following things: 
```
-  How the framework collects data from multiple sensors synchronously? For example, how can data be collected synchronously from RGBD, Pupil Core headset, and wearable sensors? How are these sensors synchronized?
- How the synchronization works? Is it offline (i.e., data are collected from multiple sensors independently and then in the post-processing step [after the completion of data collection] the data are synchronized a) or online [data are collected synchronously, and there is no need for post-procession to synchronize the data streams after the data collection]? Please explain concretely. 
- How are the online annotations performed? I would like to suggest the author prepare a short video demo of how the data labeling system works.
- During the online annotations: are the participants continuously performing the sequence of actions without any interruption from the experimenter? or an experimenter guides the participants on when to start and end one action and start another?
```
- Author claims this framework "........ is extensible to additional sensors or environments" [Line 61]. I would like to know how the researchers can extend this framework to include additional sensors. For example, how the framework ensures the synchronized data collection for the additional sensor. How easy will the extension be? Do the researchers need to change almost all the components of this framework to include additional sensors?
- The author provides the list of libraries to recreate the environment to execute the data collection framework. However, I would suggest that the author share a docker to make it easy and accessible for other researchers to recreate the environment easily.
Several multimodal activity datasets are available in the literature; how does this new dataset contribute differently to the research community? I suggest the author prepare a table to compare this new dataset to the existing and recent multimodal activity datasets, such as UTD-MHAD, MMAct, NTU-RGBD, etc.

correctness: Some of the claims are sound. I have some questions and confusion about some of the other claims which I stated in the weakness sections.
clarity: The paper is well written. But I found some missing details based on the aims of this paper. It seems to me that one of the primary contributions of this paper is the data collection framework. If it is then I would expect more concrete details about this data collection framework [please check my comments in the weakness section]
relation to prior work: I would like to know how this data collection framework differs from the existing framework. and how the collected multimodal datasets address the lacking of the existing multimodal datasets. I would like to suggest the author compare this dataset to the recent multimodal activity datasets, such as MMAct.
additional feedback: Please check my previous comments.
review_title: A multimodal dataset and synchronized multimodal data recording framework

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper presents a multimodal dataset and recording framework with an emphasis on wearable sensing in a kitchen environment. It uses a wearable sensing suite to capture motion, force, and attention information, coupled with different action sequences in the kitchen.  
strengths: The paper is the first to provide a multimodal dataset and recording framework with an emphasis on wearable sensing in a kitchen environment. It uses motion, tactile sensing, body and eye tracking, audio and image to form a dataset of kitchen activities. 
weaknesses: 3.1: The paper presents a human activity dataset with the extensible framework used to capture it. However, the selected activity in the paper is very limited (6 activities). The author claims it is a growing dataset but having 6 activities is not enough for a large dataset to produce robust results, as claimed in the Introduction. 
There is a lack of user application discussion in the paper. In particular, what use cases can be beneficial from this paper. Many kitchen activities require more information than a camera and motion, for example, temperature, shape, the texture of the different objects, etc. The author needs to provide an application that justifies the use case of what is captured in the dataset.   
correctness: The dataset is constructed correctly.
clarity: The paper is without an application part.
relation to prior work: Relation to prior works is well discussed.
additional feedback: N/A
review_title: Review of ActionNet: A Multimodal Dataset for Human Activities Using Wearable Sensors in a Kitchen Environment

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a new multimodal dataset ActionNet which records the activities of human activities in a mock kitchen environment with multiple sensors. The dataset contains multiple tasks including object manipulations, dexterous actions, and complex action sequences that are commonly seen in kitchen-related activities and all tasks are labeled by authors. The paper utilizes a large number of sensors to record the activities from the aspects of vision, sound, motion activities, and tactile feelings using wearable and external sensors.
strengths: 1. Compared to other existing activity datasets, ActionNet contains way more modalities so that it is able to capture more detailed and comprehensive information about human interaction with the environment and have a better description of human activities.

2. The dataset itself is well-organized and easily accessible. The authors provide detailed information and guidance about the information of the dataset and the user guide, which benefits the potential users.

3. The dataset may be valuable for the potential development of smart textiles and the emulation and assistance of robots for a person's action.
weaknesses: 1. The size and diversity of the ActionNet are limited compared to existing activity datasets. The ActionNet currently contains 10 subjects where each subject performs around no more than 20 activities with about 78.5 minutes of videos on average. However, the existing activity dataset, e.g. ActivityNet, contains 648 hours of videos and 200 classes, which is far more than the ActionNet. The authors claim ActionNet circumvents the issue of workspace restriction (line 38) while all videos are taken in the same mock kitchen. The paper will be much stronger if it contains more diverse scarioes and activities.

2. Lack of experiments and applications. The paper does not contain any applications or experiments that can be benefit from this dataset, which undermines the utility of this paper. Also, since the dataset contains a large amount of multimodal information, it is better to show ablation studies that how each modality is useful compared to only vision in existing activity datasets. It may raise my concern that some modalities may not be that useful in understanding human activities.
correctness: Most claims made in this paper are correct while some of them may need further evaluation, as mentioned in weakness.
clarity: The paper is clearly written and easy to understand.
relation to prior work: The authors carefully and comprehensively place their work upon existing activity datasets.
additional feedback: No additional feedbacks.
review_title: ActionNet: A Multimodal Dataset for Human Activities Using Wearable Sensors in a Kitchen Environment

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: In this paper, the authors collected a dataset with 6 subjects (25 as planned by the time it is published) on performing various kitchen cooking tasks. A number of sensor streams are captured, including motion, eye tracking, tactile sensing, EMG, body and finger tracking, audio, video, together with activity labels and subjective surveys. Subjects did a series of activities, including peeling and slicing, spreading, opening/closing a jar, wiping, pouring, and high-level tableware tasks. The authors envision that the potential usage of the dataset includes task planning, robots training, and smart textiles guiding. I appreciate the authors creating such a rich dataset with excellent documentation and decent web pages.
strengths: - A wide range of sensors and activities to provide a very rich dataset.
- The website is very well documented with clear instructions on different sensors/activities with corresponding data streamings.
- The topic can be potentially interesting to a wide range of audience.
weaknesses: - It is unclear how much impact brought by different sensors, especially gloves, on participants' natural behavior of cutting, peeling, etc. However, in order to collect the dataset, such an effect seems to be inevitable. I would recommend the authors mention it in the limitation.

- Why doesn't the activity task include some representative cooking tasks (e.g., frying, stirring, grilling, and baking, to name just a few). Please justify.

- The number of participants is never mentioned in the main text. From the website, the current dataset has 6 subjects, which is limited. However, with the pipeline established, it is not difficult to collect data from more participants. As the website mentioned, they aim for 25 subjects by 2022 Fall.

- The paper may benefit from some initial discussion of potential usage of the dataset
correctness: I don't have concerns on the correctness of the data collection process.
clarity: The paper is very clear and easy to follow.
relation to prior work: Another egocentric dataset Ego4D: Grauman, Kristen, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger et al. "Ego4d: Around the world in 3,000 hours of egocentric video." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012. 2022.
additional feedback: I want to thank the authors for doing a great job! I am looking forward to seeing the camera-ready version of the paper!
review_title: A Rich Action Dataset with Detailed Documentation

Meta-review: metareview: This paper presents an interesting multimodal dataset for research work on analyzing human actions. While there are some issues with this work, the authors have addressed most of them well during the rebuttal phase. Overall, there is sufficient support to accept this paper at this conference. The authors should include all these points in the revised paper and dataset before the conference date. 
confidence: 3: The area chair is somewhat confident
Decision: Accept