Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This dataset includes daily living action data collected by two cameras and several IMUs from 51 stroke-impaired patients and 20 healthy subjects. Different from previous human action datasets, this dataset focuses on short-duration action (functional primitives) recognition, including reach, reposition, transport, stabilization, and idle. The labeling requires a significant amount of effort from trained annotators. The motivation is to facilitate the rehabilitation process after stroke that requires repetition of sub-second actions. In addition, a new action sequence identification method is presented that can outperform other approaches and the method is validated on the presented dataset and other previous video datasets.
strengths: This dataset is different from other existing datasets and focuses on actions in short duration. This new way of action labeling provide a different view of behaviors that can be helpful for rehabilitation after stroke. With this large-scale dataset, a new action-recognition benchmark is presented using several previous segmentation-based approaches and newly developed sequence-to-sequence models inspired by speech recognition. The result also shows that the model trained on impaired patients can also be applied to healthy subjects, but the opposite is not working. And this action recognition is shown to be useful for quantifying does by counting these short-duration actions (functional primitives).
weaknesses: The paper presents a dataset including video data and motion data from IMUs. However, the motivation is not clear. What is the advantage of adding IMU sensor data? Setting up two cameras is very easy, but attaching IMUs to a human body is not very easy. It is also useful to show whether adding multiple IMUs can affect action execution. From Figure 1, it is clear that tow IMUs are attached to human hands and this setup may have some effect on impaired patients. The paper states that it introduces a multimodal dataset but the benchmark is done respectively and the paper doesn't show the advantage of the multimodal data.
correctness: Yes, the claims in the paper are correct.
clarity: The paper is well written.
relation to prior work: Yes, this paper clearly discussed the contribution compared with previous works.
additional feedback: The website of the dataset can be improved. Adding some data visualization and videos are helpful.
review_title: Short-duration action dataset collected by cameras and IMUs

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this paper, authors introduce a new multimodal dataset called StrokeRehab. It includes inertial measurements and video data of several daily activities performed by 51 stroke impared patients and 20 healthy subjects. Sequences are labeled at a high temporal resolution for annotating elemental short-duration actions. In addition, a novel approach dedicated to high resolution action identification is proposed. Experiments on the proposed dataset as well as existing ones suggest that the proposed approach is suitable for the task in comparison to state-of-the-art.
strengths: The proposed dataset is interesting and seems challenging compared to existing ones, in particular for the task of sub-second action identification. As it is multimodal, I think it could be beneficial for the research community in both the field of computer vision and time series analysis. In addition, the datasets also propose additional challenges like distributional shift and quantification of rehabilitation dose that are very interesting.
Moreover, the proposed approach for high resolution action identification sounds good and promising.

weaknesses: The fact that authors simultaneously propose a new dataset and a novel approach can also be seen as a weakness. Indeed, it results in partial information for both parts. While the supplementary material is significant and provides useful information, the necessity to make back-and-forth between the paper and the appendix does not facilitate the reading and the whole understanding. It would maybe be a good idea to focus the paper on the dataset (with a benchmark using existing methods) as it is the scope of this track, and let the novel approach for a further paper.
correctness: For the dataset part, it seems that it has been constructed in a sound way.
For the benchmark part, I think the robustness could be improved regarding the train/test split. Indeed, If I understood well, either for healthy or stroke patients, a single train/test split has been considered. As a result, the compared approaches are evaluated on this particular split. Hence, it is difficult to conclude about the reported performances, if they are general or valid only for this particular case. I think considering different splits in a cross-validation scheme would add significant robustness. A common scheme in similar cases is the use of the one-subject-out cross validation.
clarity: The paper is clear but could be better organized. Moreover some typos and grammatical errors remain.
relation to prior work: The differences between the proposal and previous contributions are mentioned. However, most of these discussions are included in the appendix and it does not make the reading smooth.
additional feedback: Please find below some additional minor remarks/suggestions:
- I suggest authors carefully read the paper to correct some remaining typos
- First sentence of the abstract and the introduction are the exact same
- I think subsection 2.3 can be merged with subsection 2.1
- Table 3 should be Figure 3 as it shows confusion matrices
review_title: StrokeRehab dataset and sequence-to-sequence approach for sub-second action identification

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This article proposes a dataset for identifying short duration actions for physical rehabilitation that contains both video and IMU data. THe dataset includes data from stroke-impaired patients and healthy subjects performing activities of daily living, and labels at a high temporal resolution.

Along with the dataset, the article proposes a novel sequence-to-sequence approach to identify actions, and presents the perfomance on this dataset and other datasets of two versions of algorithm from their proposed approach and compared with state-of-the art algorithms.

strengths: * The dataset proposed contains data from a large number of patients.
The dataset has also been annotated by several people, following a precise protocol, with a good interannotator agreement score.

* The algorithms proposed are contrasted against state of the art algorithms on both the proposed dataset and other datasets
weaknesses: * the annotators training and annotation protocol is very precise but I have doubts about its adequacy, the process needs to be justified. The training seems to enable all annotators to acquire the same definition of the actions and the labeling process. However, I have concerns this lengthy process also erases the inter-annotator variability : the "coders" could end up emulating the expert's annotation instead of labeling by his own judgement, and this would erase inter-annotator variability. The labeling by experts would be more interesting than by coders, even with a smaller number of labelers.

* the approaches proposed lack a detailed description in the article. A high-level explanation and motivation are given, but the final algorithms themselves are not described in the article, only in the supplementary material.

* the dataset aims at segmenting elementary actions that are quite similar. However, the data acquisition uses wearable sensors that are placed on the hands : they and the straps can be bulky and change the natural motion of the subjects. This change can have an impact especially on similar movements. Has this impact been evaluated ?






correctness: * In table 2, the model trained on SP has the following results : the accuracy of the test on healthy subjects is better than tests on stroke patients. This seems strange that the model performs worse on the distribution it was trained on. This should be addressed, and does not show the distributional shift, as stated in the article
Moreover, the model trained on HS+SP has better results than the model trained only on SP when tested only on SP. Again, this can seem puzzling and should be addressed, as this is not a sign of the distributional shift described in the article.

* In section 4.2, l. 271:"For the video data, confusion occurs mainly between primitives which are usually performed one after the other "  :it does not sound a convincing hypothesis, it seems based on only 1 example. What is the usual order of actions ?

* In table 3, the confusion matrices do not seem to be normalised, neither by row nor by column.

* there are a few errors in the references, for instance "Table 4.3"
clarity: * I did not find in the article the number of annotators.
* In section 2.4, what is the definition of "severely impaired" subjects ? How is the severity assessed, and which is the threshold ?
* what does “removing duplicates at the boundaries“ mean ?
* the word "action" is used in this article with several meanings: from movements or simple actions (reaching, transporting) to activities or complex actions (brushing,feeding). But the wording is not always clear which "action" it is currently about.

relation to prior work: * The authors argue that existing datasets do not label short-duration actions. However, the number of datasets of actions using IMU and/or videos is very high. Therefore, a more precise definition of the subfield to compare to, would have been necessary. Currently, the listed datasets are very few compared to the state of the art. Moreover, I feel that a focus could have been developed on medical datasets for physical rehabilitation.

additional feedback: none
review_title: Micro-action dataset for activity of daily living rehabilitation 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper describes the StrokeRehab Dataset for action sequence recognition. The dataset consists of sub-second 3,372 trials (120,891 functional ‘primitives’) in 71 subjects. Data was recorded with 9 IMUs on the upper extremities, and featurized using the IMU acceleration, quaternion output, and joint angle representation from a proprietary algorithm. Video data was also recorded from two views as well. The data was labeled in a highly granular manner, under supervision of a domain-expert. Using this dataset, the authors tested different approaches for action sequence recognition, finding sequence-to-sequence models outperformed segmentation based models for this data as well as others in the field. 

Overall I was impressed by the granularity of action labeling in the paper and the novelty of the application to the community. I think the authors make interesting and likely valid conclusions about the utility of action segmentation vs. seq-2-seq approaches for action labeling for short primitives, which speaks to the potential for the reuse of the dataset in the community. So I am open to accept this manuscript, however I have reservations below that may affect this decision. While I was impressed by the annotation density, I do have multiple questions that I would like to be resolved before acceptance. 

strengths: The paper contextualizes its results well in the field and is well described. 
weaknesses: While I believe the dataset is largely sound, I have multiple comments that I would like to see clarified and resolved. 

### Data Collection

•	Were the video and sensor data synchronized?

•	Can you comment on how similar the video and sensor data were from subject to subject and day to day. Knowledge of domain differences across video recordings (e.g. shifts of perspective) will affect their generalizability. Raw IMU data will depend on the subject size. 

### Rehabilitation Applications 

•	I was confused about how AER was chosen as the metric of choice, as opposed to TPR/FDR of actions, or more general task-based quantities (eg effector velocity). The description is cursory, and it is important to communicate for non-domain experts. It also affects the conclusions, as segmentation models were more performant on the basis of TPR. 

•	Is the accuracy of these models sufficient to reach desired capacity for physical therapy applications? Much of the scope of this application track is so domain-agnostic experts , and it is not clear what AER is satisfactory for applications.

•	Related to the above: what is the inter-human reliability in the AER?

•	L295 “detailed count-based results are provided in Appendix D” I am not sure what this refers to. 

### Domain shift: 

* Why is there asymmetry in the transfer of models in Table 1? Why does stroke data generalize to health but not vice versa? One trivial reason is that because there are more users in the stroke dataset, they may ‘cover’ domains better, e.g. contain participants of a greater range of sizes and shapes and video conditions, improving the machine learning transfer. Were the comparisons in Table 1 balanced?

* Can you comment on how labels were made across domains, that is, were different definitions based e.g. on kinematics made for labeling stroke primitives given that they may be overtly different kinematically

* What is the quality of action recognition models on examples from this dataset, e.g. given individual pre-segmented videos? Is the issue with the seq-seq models and segmentation models that actions are hard to recognize or that they are hard to segment?

* Table 3: Why don’t confusion matricies sum up to one row-wise or columnwise to 1? Also it should be clarified whether the matricies are normalized row-wise or column-wise. 


correctness: I think the dataset methodology and benchmarking is largely sound, although see specific points above. 
clarity: Yes.
relation to prior work: Yes.
additional feedback: None.
review_title: Detailed annotations for sub-second action sequence recognition

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: A new dataset called StrokeRehab provides action information for multiple patients rehabilitating from strokes as well as healthy subjects performing similar actions. The dataset contains information from multiple modalities (sensors and video data). The paper also proposes a new algorithm to improve action recognition on this dataset.
strengths: 1. The motivation of the work is clear. Such a dataset can help the medical community in multiple ways and design better rehabilitation programs in the future.

2. While the absolute number of identities present in the dataset (patients and healthy subjects) is small, I believe the dataset is sufficiently large in the medical domain.  

3. The authors manually label the data and cross-check the quality of labels. 

4. The authors first show that naively training the current state-of-the-art approaches on this dataset leads to inferior results and proposes a new approach to tackle the task.


weaknesses: 1. I did not find Figure 6 in the paper. I believe it will be the figure which has the architectural definition. The figure is in the supplementary material and should be mentioned properly in the main paper. I believe this is an important figure and thus should be included in the main paper.

2. I believe transformers can also be tried instead of LSTM-based architectures. Data scarcity might be a valid reason not to use transformers which are known to be data hungry, but this should be clearly mentioned and discussed in the paper.

3. While a large amount of sensor data is collected, the information provided in the paper regarding this is less. I believe more information like where the sensors are placed can be provided.








correctness: I found the claims to be correct
clarity: The paper writing can be improved a bit. The figures are too small and often not referred to properly.
relation to prior work: The authors explain prior works and datasets properly.
additional feedback: None
review_title: Review for StrokeRehab: A Benchmark Dataset for Sub-second Action Identification

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The authors introduce a large-scale action-recognition dataset called StrokeRehab. The contributions of this paper are summarized as follows:
1. This dataset provides a benchmark for sub-second action identification.
2. The dataset consists of 3,372 trials of rehabilitation activities and 120,891 functional primitives.
3. It provides a benchmark for generalization in the presence of realistic distributional shift.
4. It can be used for quantifying dose by counting functional primitives.

strengths: 1. This dataset consists of a considerable amount of short-duration elemental actions, which required more manual effort to label and more time for training.
2. This dataset contains both video and sensor data, which can leverage both modalities to perform the task of sequence estimation.
3. This dataset provides a benchmark for generalization in the presence of realistic distributional shift and data-driven quantification of rehabilitation dose.

weaknesses: 1. It seems not novel to make a dataset for short-duration action identification since there are many similar practices.
2. There are some writing mistakes, e.g.:
In Table 2, "on both (HS)" should be "on both (HS + SP)". 
3. The experiment part is relatively insufficient. The author should supplement the elaboration of the experimental settings, such as the selection of the backbone of the baseline models used, the implementation details of training and testing, etc.
4. Lacking experiments to verify that this dataset can improve the performance of identifying more elemental motions at high temporal resolution. The models trained on this dataset should be tested on the existing datasets and the real scenes. And it would be better to supply the visualized results.
5. The authors spend a little too much space on the action sequence identification algorithm, but the proposed dataset is the theme. To highlight the content of the dataset, the structure of the article should be adjusted appropriately.

correctness: The way to construct the dataset sounds appropriate. 
clarity: No, there are some writing mistakes, and the structure of the article should be adjusted appropriately. 
relation to prior work: No, some works, e.g. [R1], have the similar practice to this work. Besides, the sub-second actions in the dataset emphasized by the authors seem to have little significance for improving the performance of the model.
[R1] H. Kuehne, A. Arslan, and T. Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 780–787, 2014.

additional feedback: In Weaknesses, I described in detail what can be improved. The authors can check my comments in Weaknesses for improvement.
review_title: This paper proposed a new dataset for sub-second action identification. The dataset consists of a considerable amount of short-duration elemental actions. However, the idea of recognizing actions with high temporal resolution is not novel. Moreover, the writing is not very good, and the experiments are not enough.

Meta-review: metareview: This submission receives reviews from 6 different reviews. Most reviewers (5/6) appreciate the contribution of the new dataset. They acknowledge that the problem setup is interesting and the dataset may be useful for different research communities: computer vision, time series analysis, medical. On the other hand, reviewer 38gN concerns about the novelty of the proposed dataset. AC reads all reviews and comments, and discussions, and is convinced that the proposed dataset will provide a useful benchmark for research, thus AC recommends to accept this submission as a poster. AC recommends the authors to incorporate all suggestions from reviewers for the final camera version.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept