SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Insight #3: Null action understanding requires efforts on addressing imbalanced class distribution. Table 9 shows the recall and precision of action recognition and action segmentation for null actions. We suspect the high recall and low precision is caused by the imbalanced class distribution, as null is the largest head class (see Figure 5). To address the class imbalance problem, we randomly over-sample the minority classes (classes with sample size below a threshold) to reach the threshold, and report the action recognition accuracy of UniFormerV2 on the over-sampled dataset in Table 10. Here, the threshold is set to 300 and more details can be found in the Supplementary.
Table 9: Recall and precision of null recognition and segmentation. Action recognition results are from MVITv2 and action segmentation results are from BCN.
Recall	Precision
Recognition	Primitive Task	90.8%	65.1%
Atomic Action	81.5%	39.1%
Segmentation	Primitive Task	80.9	45.1%
Atomic Action	84.6%	37.5%
Table 10: The action recognition accuracy of UniFormerV2 on the over-sampled dataset.
Method	View	Primitive Task	Atomic Action
Left-hand	Right-hand	Left-hand	Right-hand
Top-1	Top-5	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
UniFormerV2	Average	62.4	89.7	61.4	89.9	48.9	80.9	44.6	77.6
UniFormerV2 (over-sampling)	Average	67.3	89.2	66.2	89.1	52.0	81.2	48.8	78.1
Insight #4: New research from wrong action annotations. Wrong actions refer to assembly actions (primitive task level) occurred at the wrong position or order. Our annotated wrong actions can initiate three avenues for research. First, the pattern of wrong actions in different participants across the three-stage progressive assembly can provide insights into human learning progression and performance. Second, investigating the wrong action and the actions leading up to it can help identify assembly errors, contributing to quality assurance. Third, analyzing the actions undertaken after the wrong action can provide insights into how humans resolve errors and re-plan the assembly sequence.
3.4 Task Collaboration
Understanding the states, patterns, and dynamics of two-handed collaboration during the assembly process is essential for applications, such as ergonomics analysis, collaborative task planning, and human-robot collaboration design. HA-ViD can support research in this aspect via providing spatial annotations, two-hand separated temporal annotations and collaboration status annotations.
Insight #5: New research on understanding parallel tasks from both hands. Table 11 shows that both action recognition and segmentation have lowest performance on parallel tasks during assembly. One possible reason is that the foundational video understanding methods rely on global features of each image, and do not explicitly detect and track the action of each hand. This calls for new methods to independently track both hands and recognize their actions through local features. Recent research on human-object interaction detection in videos [36, 37] could offer valuable insights.
Table 11: Recall of two-handed primitive task recognition and segmentation in four collaboration status. Action recognition results are from MVITv2 and action segmentation results are from BCN.
Action recognition results	Action segmentation results
Collaboration	Parallel	Single-handed	Pause	Collaboration	Parallel	Single-handed	Pause
Left hand	52.5%	39.7%	54.2%	92.4%	32.1%	15.4%	18.5%	85.5%
Right hand	46.1%	30.5%	50.7%	93.3%	35.0%	24.2%	17.2%	82.9%
3.5 Skill Parameters and Human Intention
Understanding skill parameters and human intentions from videos is essential for robot skill learning and human-robot collaboration (HRC) [38, 39].
Typically, skill parameters vary depending on the specific application. However, there are certain skill parameters that are commonly used, including trajectory, object pose, force, and torque [40, 41].
While videos cannot capture force and torque directly, our dataset offers spatial annotations that enable tracking the trajectory of each object. Additionally, the object pose can be inferred from our dataset via pose estimation methods. Therefore, HA-ViD can support basic research in this direction. To acquire more detailed and accurate skill parameters, assembly process comprehension must extend to the 3D space, utilizing 3D reasoning and 3D hand-object interaction estimation techniques to precisely track the temporal changes in hand poses and objects. For human-robot skill transfer, the learned assembly skill parameters can be sourced as input to robot learning environments.
Understanding human intention in HRC refers to a combination of trajectory prediction, action anticipation and task goal understanding [42]. Our spatial annotations provide trajectory information, SA-TPGs present action sequence constraints, and GAB CAD files offer the final task goals. Therefore, HA-ViD can enhance the research in this aspect.
As identified in Section 3, HA-ViD provides a basis for developing video understanding and reasoning techniques to derive application-oriented knowledge. Model design, results analysis, and knowledge reasoning pipeline development that are based on HA-ViD can accelerate the development of application-specific models. In addition, if a new dataset is required for a specific application, our HR-SAT-based annotation protocol can be employed to ensure the resulting target dataset is compatible with HA-ViD. The annotation alignment eases the process of adapting and deploying pre-trained models to new scenarios.
We acknowledge the limitation of HA-ViD on fully capturing the complexities and diversities of industrial assembly scenarios. Therefore, HA-ViD can be extended to include more products, assembly environments, and even different agents. Following our data collection and annotation protocol could ensure similar high-quality assembly video datasets that are compatible with HA-ViD.
Furthermore, we identify that HA-ViD could benefit from 3D and pixel-wise geometric annotations. This can facilitate the research into 3D hand-object interaction and 3D scene understanding, which is essential for assembly quality checking and robot skill learning. Therefore, future work could involve providing more refined annotations, such as hand poses, object poses and 3D key points, via additional sensors. We therefore created a dataset roadmap on our Github repository to outline improvement focuses and encourage collective efforts from the community to extend HA-ViD.
We present HA-ViD, a human assembly video dataset, to advance comprehensive assembly knowledge understanding toward real-world industrial applications. We designed the Generic Assembly Box to represent industrial assembly scenarios and a three-stage progressive learning setup to capture the natural process of human procedural knowledge acquisition. The dataset annotation follows the Human-Robot Shared Assembly Taxonomy. HA-ViD includes (1) multi-view, multi-modality data, fine-grained action annotations (subject, action verb, manipulated object, target object, and tool), (2) human pause and error annotations, and (3) collaboration status annotations to enable technological breakthroughs industrial application-oriented knowledge comprehension from videos.
We benchmarked strong baseline methods of action recognition, action segmentation, object detection and multi-object tracking, and analyzed their performance and the further reasoning steps for comprehending application-oriented knowledge in assembly progress, process efficiency, task collaboration, skill parameter and human intention. The results show that our dataset captures essential challenges for foundational video understanding tasks, and new methods need to be explored for application-oriented knowledge comprehension. We envision HA-ViD will open opportunities for advancing video understanding techniques to enable the futuristic ultra-intelligent industry.
This work was supported by The University of Auckland FRDF New Staff Research Fund (No. 3720540). Our gratitude extends to the participants and annotators of our dataset for their essential contributions. The Industrial AI Research group has provided invaluable feedback and advice, with
particular thanks to Benedict Liang for benchmarking assistance, and Saahil Chand for his annotations and insights.
[34] D. Gong, J. Lee, M. Kim, S. J. Ha, and M. Cho, “Future Transformer for Long-term Action Anticipation,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3042–3051, IEEE, jun 2022.
[35] Y. A. Farha, A. Richard, and J. Gall, "When will you do what? - Anticipating Temporal Occurrences of Activities," in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5343-5352, IEEE, jun 2018.
[36] D. Tu, W. Sun, X. Min, G. Zhai, and W. Shen, "Video-based Human-Object Interaction Detection from Tubelet Tokens," in Advances in Neural Information Processing Systems 35, pp. 23345--23357, 2022.
[37] M.-J. Chiou, C.-Y. Liao, L.-W. Wang, R. Zimmermann, and J. Feng, “ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videos,” in Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval, (New York, NY, USA), pp. 9–17, ACM, aug 2021.
[38] O. Mees, M. Merklinger, G. Kalweit, and W. Burgard, "Adversarial Skill Networks: Unsupervised Robot Skill Learning from Video," in 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4188-4194, IEEE, may 2020.
[39] P. Zheng, S. Li, L. Xia, L. Wang, and A. Nassehi, “A visual reasoning-based approach for mutual-cognitive human-robot collaboration,” CIRP Annals, vol. 71, no. 1, pp. 377–380, 2022.
[40] J. Jeon, H.-r. Jung, F. Yumbla, T. A. Luong, and H. Moon, "Primitive Action Based Combined Task and Motion Planning for the Service Robot," Frontiers in Robotics and AI, vol. 9, feb 2022.
[41] E. Berger, S. Grehl, D. Vogt, B. Jung, and H. B. Amor, "Experience-based torque estimation for an industrial robot," in 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 144-149, IEEE, may 2016.
[42] Y. Lu, H. Zheng, S. Chand, W. Xia, Z. Liu, X. Xu, L. Wang, Z. Qin, and J. Bao, "Outlook on human-centric manufacturing towards Industry 5.0," Journal of Manufacturing Systems, vol. 62, pp. 612-627, jan 2022.