HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understanding
Authors: Hao Zheng
Regina Lee
Yuqian Lu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Assembly video dataset
Video understanding
Assembly knowledge understanding

Abstract: Understanding comprehensive assembly knowledge from videos is critical for futuristic ultra-intelligent industry. To enable technological breakthrough, we present HA-ViD – the first human assembly video dataset that features representative industrial assembly scenarios, natural procedural knowledge acquisition process, and consistent human-robot shared annotations. Specifically, HA-ViD captures diverse collaboration patterns of real-world assembly, natural human behaviors and learning progression during assembly, and granulate action annotations to subject, action verb, manipulated object, target object, and tool. We provide 3222 multi-view and multi-modality videos), 1.5M frames, 96K temporal labels and 2M spatial labels. We benchmark four foundational video understanding tasks: action recognition, action segmentation, object detection and multi-object tracking. Importantly, we analyze their performance and the further reasoning steps for comprehending knowledge in assembly progress, process efficiency, task collaboration, skill parameters and human intention. Details of HA-ViD is available at: https://iai-hrc.github.io/ha-vid.

HA-ViD: A Human Assembly Video Dataset for
Comprehensive Assembly Knowledge Understanding
Hao Zheng hzhe951 anduni ac.nz
Regina Lee"
Xlee7020auck anduuiiciinnniiiiii anduni ac.
Yuqian Lu
yuqian lu@auckland ac. nz
Department of Mechanical and Mechatronics Engineering The University of Auckland
Abstract
Understanding comprehensive assembly knowledge from videos is critical for futuristic ultra-intelligent industry. To enable technological breakthrough, we present HA-ViD assembly video dataset that features representative indus- trial assembly scenarios, natural procedural knowledge acquisition process, and consistent human-robot shared annotations. Specifically, HA-ViD captures di- verse collaboration patterns of real-world assembly, natural human behaviors and learning progression during assembly, and granulate action annotations sub- ject, action verb, manipulated object, target object, and tool. We provide 3222 multi-view and multi-modality videos, 1.5M frames, 96K temporal labels and 2M spatial labels. We benchmark four foundational video understanding tasks: action recognition, action segmentation, object detection and multi-object tracking. Importantly, we analyze their performance and the further reasoning steps for comprehending knowledge in assembly progress, process efficiency, task collabo- ration, skill parameters and human intention. Details of HA-ViD is available at: wwt.thhrrr.gg.b.b.......g... attps::/iiae hrc. github. io/ha-vis
Introduction
Assembly knowledge understanding from videos is crucial for futuristic ultra-intelligent industrial applications, such as robot skill learning [], human-robot collaborative assembly [2] and quality assurance [3]. To enable assembly video understanding, video dataset is required. Such video dataset should (1) represent real-world assembly scenarios, (2) capture the comprehensive assembly knowledge, (3) follow a consistent annotation protocol that aligns both human and robot assembly comprehension. However, existing datasets fall short in meeting all these requirements.
First, the assembled products in existing datasets either highly application-specifi 4 lack representative assembly parts and tools (77 Second, many datasets did not design assembly tasks to foster the emergence of natural behaviors (e.g., varying efficiency, alternative routes, pauses and errors) during procedural knowledge acquisition. Third, thorough understanding of nuanced assembly knowledge is challenging with existing datasets they often do not annotate subjects, objects, tools and their interactions a systematic approach.
Therefore, introduce HA-ViD: human assembly video dataset recording people assembling the Generic Assembly Box (GAB, see [T. We benchmark on four foundational video understanding
"These authors contributed equally to this work. Conresponding author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

tasks: action recognition, action segmentation, object detection and multi-object tracking (MOT), and analyze their performance and the further reasoning steps for comprehending application-oriema oriented knowledge. HA-ViD features three novel aspects:
Representative industrial assembly scenarios: GAB includes 35 standard and non- standard parts frequently used in real-world industrial assembly scenarios and requires 4 standard tools to assemble it. The assembly tasks are arranged onto 3 plates featuring different task precedence and collaboration requirements to promote the emergence of two-handed collaboration and parallel tasks. Compared existing assembly video datasets, GAB is more representative of generic industrial assembly scenarios (see Table
Natural procedural knowledge acquisition process: Progressive observation, thought and practice process (shown as varying efficiency, alternative assembly routes, pauses, and errors) in acquiring and applying complex procedural assembly knowledge is captured via the designed three-stage progressive assembly setup (see Figure [T). This design allows in-depth understanding of the human cognition process, where existing datasets are limited
(see Table
Consistent human-robot shared annotations: We designed a consistent fine-grained hierarchical task/action annotation protocol following Human-Robot Shared Assembly Taxonomy (HR-SAT) to be introduced in Section 2.3). Using this protocol, we, (1) granulate action annotations subject, action verb, manipulated object, target object, and tool; (2) provide two-handed collaboration status annotations; and (3) annotate human pauses and errors. Such detailed annotation embeds more knowledge sources for diverse understanding of inplicatiio-orientte knowledge (see Table T
Apsentby
Dataset Requirements
Systeman'ss anroiiber proboo!
Noruevobbr compottruicss
Inketty
efieeeny aberaativ rodtrs ewar anat
Pause and scoration
Propees Eleannc
Applications
Treit Collsbonation
Colebastiv ano parllll tutk real.
Progenaie hariing d'asseely
File goaney tempoos and seahel
Robor skil eamiing
Represer ladive industrial Apuerttly Scenario
Natural Procedural
Consistent llumnn-Rooot
Human ribot cokstoration
Know Auquasition
Stared Amnosatione
Acsembly process monitring
rssembiy tock paanning
Quallty by assurance
HA-VID
Ski Paramiters stars
Human intertioon
Aasembly Product
Data Collection
Ceneric Assemnly Box
Null-siee
S-stage prograssive issembly
Cophootior-eriettee Kiowllagge
disc, foot, bop
Intouderr
pirtes diferent
diandard
Kudk by
Trabucher"
Dealld
took orecedienon and cxlabboratoon regu
RGB Dupais
No
Assembly Video Ressoning
Data Annotation
Temporrd Annotatone
Hurran Robot
Latkhard
Righislend
Sharet hosembb
Understanding Techniques
Foundational Video
Prrnt se Taok
Soow stull veo Folo7
Nal
Issent stae
Arosng
Piinivve Task
Aperoech pear Il Greep grvw
Move geor N lassn pear onto Stot
Action
Action Recogrition
Atamic Action
ArtiicArtion
Segmentation
Colluto Stalus
Object Mult-objjct
Cabiteootoon
Pleua
Shope-bandee
Pars
Placse
Oijects,
Racogniton
Tracking
Figure 1: HA-ViD, dataset designed for industrial applications, represents real-world assembly scenarios, and captures the process of acquiring procedural knowledge. The consistent annotation follows human-robot shared taxonomy. The dataset features 3222 multi-view and multi-modallties videos, 1.5M frames, 96K temporal labels and 2M spatial labels.
2 Dataset
In this section, we present the process of building HA-ViD and provide essential statistics.
SAT, developed by the same authors, is a hierarchical assseblll task representation schema that both humans and robots can comprehend. See details via: https:///ai github. -ithr-sat

---

Table 1: Comparison between HA-ViD and other assembly video datasets.
Dhtoeet
Assembled
product
Varying assemilty
efficiency
knowledge aquisition process
Fusural peocedual
Aliemaatv
toute
Parse
Frnur
Subjezt
Action vesb
shared assemblly tasenotty
Consistent haman robor
Manapulsted
ebject
abject
Targer
Tool
Two-hund
collabocation
Too-handed
statis
Monden Box
Wooden Box
IKEA
Furniare
MECCANO
Toy motorbike
IKEA ASM
Funiine
Assemblyy10 101
Tay
x
HA4M 4
Gear Trui
Epizyelic
V
HA VID
(ours)
sesetbll bus
Generic
/
/
V
2.1 Generic Assembly Box
To ensure the dataset can represent real-world industrial assembly scenarios, we designed the GAB shown in Figure !
First, GA is 250 2550x250xx2mm) 250mm box including 11 standard and 24 non-standard parts frequently used real-world industrial assembly. Four standard tools are required for assembling GAB. The box design also allows participants naturally perform tasks top side-facing plate, closer to the flexible setups of real-world assembly.
Second, GAB consists of three plates featuring different task precedence and collaboration require- ments. Figure 2 shows the subject-agnostic task precedence graphs (SA-TPG) for the three plates with different precedence constraints. These different task precedence graphs provide contextual links between actions, enabling situational reasoning with different complexities. The cylinder plate also has more collaboration tasks, posing greater challenges for understanding collaborative assembly tasks. Gear and cylinder plates contain parts that become hidden after assembly, e.g., spacers under the gears. This introduces additional complexities for understanding assembly status.
General Plate
Rudio] (rdeec1)
Scrww Präpptao)
Baltsb)
Hidd
fg
md
IT
Nunt
USB komma
Move
Fand-
Holoj44)
Brolbe
Wheel (hay
Unea bournngl))
Screw
Gear Plate
Nuthn)
Placa
space
Small gconge)
Large goarlg
bst
oYthen
rgH
Shorit)
Large spaserto0
Shaftt)
Bawowkhvvr
poctp)
Bookci
row
Sppo3dp
Hoeigh)
(owg))
gastljp)
Screediver Phi
Cylinder Piate
I0400
Cyliner
shhchth
Scow heopet)
basa(c))
Annotetion specification
otho3th
Brooketiok)
heaten)
Screw
db
sholth
Cytider
Hroejc Eoxtr)
captes)
Action
Marsodllted
Target dbject
precedence
coltoorc
Holejo4)
Figure 2: Subject-agnostic task precedence graphs for three plates and annotation specification. "muus-collaaoorte" denotes the task requires two-handed collaboration, and "tend-to-collabortt" denotes the task that tend to need two hands.
2.1.1 Dataset Collection
Data was collected on three Azure Kinect RGB+D cameras mounted to assembly workbench facing the participant from left, front and top views (see Figure Videos were recorded at 1280 720 RGB resolution and 512 depth resolution under lab lighting and natural lighting conditions. 30 participants (15 male, 15 female) assembled each plate to 12 times during a 2-hour session.
FFid GAB CAD files at: https: /tei:///aiiiii wittee.ittubbb.. testthu..oohhaa... hoohhaviid vid
3

---

To capture the progression of human procedural knowledge [0]] acquisition and behaviors (e.g.. varying efficiency, alternative routes, pause, and errors) during learning. ttree-stage progressive assembly setup is designed. Inspired by discovery learning [I]] we design the three stages a Discovery- - participants are given minimal exploded view instructions of each plate; Instruction participants are given detailed step-by-step instructions of each plate; Practice participants are asked to complete the task without instruction.
The first stage encourages participants to explore assembly knowledge to reach goal, the second stage provides targeted instruction to deepen participants' understanding, and the last stage encourages participants to reinforce their learning via practicing. During Instruction and Practice stages, the participants were asked to perform the assembly with the plate both facing upwards and sideways.
2.1.2 Dataset Annotations
We provide temporal and spatial annotations to capture rich assembly knowledge shown in Figure 1
To enable human-robol assembly knowledge transfer, annotations follow the HR-SAT (shown in F.. According to the HR-SAT, an assembly task can be decomposed into primitive tasks and further into atomic actions. Each primitive task and atomic action contain five description elements: subject, action verh, manipulated object, targel object and tool. Primitive tasks annotations describe a functional change of the manipulated object, such inserting gear on shaft or screwing a nut onto a bolt. Atomic actions describe an interaction change between the subject and manipulated object such as hand grasping the screw or moving the screw. HR-SAT ensures the transferability, adaptability, and consistency of annotations. It can be used to transform annotations from other datasets into our designated description, or annotate new data to extend HA-ViD.
Tinkk
persterr
Spic
WrS.00017
Primtid
Alomm Aztion
Clial
Oloect
Matea
Halter anl Via daptonakte
Figure 3: Human-robot shared assembly taxonomy (HR-SAT) schema. We tailored the original taxon- omy by removing information that cannot be annotated from videos and incorporating a Disassemble action verb to describe human error-and-correctoon process. We provide textual annotations (see Figure 2 following the typical input formats of current video understanding algorithms. We also offer SA-TPGs as knowledge graphs n RDF/XML format following the HR-SAT schema to enable advanced assembly knowledge reasoning with enhanced relationship information.
We annotate human pause and error as null and wrong respectively to enable research on understanding assembly efficiency and learning progression. Our annotations treat each hand as a separate subject. Primitive tasks and atomic actions are labeled for each hand to support multi-subject collaboration related research. Alongside the primitive task annotations, we annotate the two-handed collaboration status as: collaboration, when both hand work together on the same task; parallel, when each hand is working a different task; single-hane handed, when only one hand is performing the task while the other hand pauses; and pause, when neither hand is performing any task.
For spatial annotations, we use CVAT video annotation tool, to manually label bounding boxes for subjects, objects, and tools frame-by by---b---eeeeeee Furthermore, we treat important assemblable features
'The instruction files can be found at //tttp::///iat htte://r//.iiiieeeeeeeeeee tithb.. io/ha- The detailed instructions were developed following HR-SAT to align assembly instructions with our annotations. 'The ST-TPGs files can he downloaded at: http::/iaii-he hww..ittub..i//...... https:/ :/www..//wwww./// evat. ai/

---

(such as holes, stud and USB female) as objects, to enable finer-graine grained assembly knowledge
understanding.
2.2 Statistics
In total, we collected 3222 videos with side, front and top camera views. Each video contains one task the process of assembling one plate. Our dataset contains 86.9 hours of footage, totaling over 1.5 million frames with average of min 37 sec per video (1456 frames). To ensure annotation quality, we manually labeled temporal annotations for 609 plate assembly videos and spatial annotations for over 144K frames. The selected videos for labeling collectively capture the dataset diversity by including videos of different participants, lightings, instructions and camera views.
Figure 4: Side, front and top camera views of the workbench.
Overall, dataset (in Fig. 5 contains 18831 primitive tasks across 75 classes, 63864 atomic actions across 219 classes, and close to 2M instances of subjects, objects and tools across 42 classes Our dataset shows potential for facilitating small object detection research 46.6% of the annotations are small objects.
231 2013 (TE (PN)
Total 1,966285
920.227
(48.8%)
33 402 Lac-tand Atomic
Prinkka
Lift
13,338 Arronaaara
46,622 Action Ar Artutioon
$96.603 8%)
9. 9,231 Rightaadd Areuii
75: SaMe Cood bas
Small: emaller than 32" px
821px 96
Larget: lorger than 90 px
(b)
Figure 5: Temporal and spatial annotation statistics. (a) Total number of temporal annotations and annotation distributions, categorized by hands. The three head classes of primitive tasks and atomic actions are shown. (b) Total number of spatial annotations categorized into COCO object scale.
Our temporal annotations can be used to understand the learning progression and efficiency of participants over the designed three-stage progressive assembly setup. The combined annotation of wrong primitive task, pause collaboration status and total frames can indicate features such as errors, observation patterns and task completion time for each participant. Our dataset captures the natural progress of procedural knowledge acquisition, as indicated by the overall reduction in task completion time and pause time from stage I to 3, as well as the significant reduction in errors (see Figure 6p. The wrong and pause annotations enable research on understanding varying efficiency between participants.
By annotating the collaboration status and designing three assembly plates with different task precedence and collaboration requirements, HA-ViD captures the two-handed collaborative and parallel tasks commonly featured in real-world assembly, shown in Figure 7 Overall, 49.6% of the annotated frames consist of collaborative or parallel tasks. The high percentage of two-handed tasks enables research in understanding the collaboration patterns of complex assembly tasks.
5

---

lotal frames frames
frames
9608
M000
farres
20.8
1906
W000
1909
0006
60
1308
1526
204
200
508
$10
611
512
513
514
B15
516
0.00 5117
Insmaationn
Partcipant ID
Slage
Stago
(a)
(6)
(C)
Figure 6: Annotation statistics of total frames, pause frames, and wrong frames. (a) Total frames and pause frames distribution by participant. (b) Average total frames and pause frames per task in each progressive assembly stage. (c) Average wrong frames per task in each progressive assembly stage.
General plate
Gear plate
Cylinder plate
17.6%
21 0%
35.0%
31 1%
$5. 9%99%
8
36.1%
34.4%
14,2%
Collaboration
Single-lended
Panalel
Pauar
Figure 7: Percentage distribution of each collaboration status annotation for each assembly plate.
3 Benchmark Experiments
We benchmark SOTA methods for four foundational techniques for assembly knowledge understand- ing, i.e., action recognition, action segmentation, object detection, and MOT. Due to the page limit, we highlight key results and findings in this section, and present implementation details, more results and discussions in the Supplementary document.
3.1 Action Recognition, Action Segmentation, Object Detection and MOT
Action recognition is classify sequence of video frames into an action category. We split 123 out of 609 temporally labeled videos to be the testset, and the rest as trainset. We benchmark seven
action recognition methods from three categories: 2D models (TSM 112, TimeSFormer (I3),, 3D)
models (I3D [14, MVITv2 [15], UniFormerV2 [16]), and skeleton basdddbaedddee methods (ST-GCN (17). ST-GCN++ [8] and report the Top-1 accuracy and Top-5 accuracy in Table
Action segmentation is to temporally locate and recognize human action segments in untrimmed videos [19]. Under the same train/test split action recognition, benchmark four action seg- mentation methods, MS-TCN [20]. DTGRM [19], BCN [21], and C2F-TCN [22], and report the frame-wise accuracy (Acc), segmental edit distance (Edit) and segmental F1 score at overlapping thresholds of 10% in T
Object detection is detect all instances of objects from known classes 24 We split 18.4K out of 144K spatially labeled frames to be the testset, and the rest as trainset. We benchmark classical two-stage method FaNterRCNN (24). one-stage method Yolov5 (25). and the SOTA end-to-end Transformer-basee based method DINO (26] with different backbone networks, and report the parameter size (Params), average precision (AP), AP under diffcrent IoU thresholds (50% and 75%), and AP under different object scales (small, medium and large) Table 4
MOT aims at locating multiple objects. maintaining their identities, and yielding their individual trajectories given an input video 27). We benchmark SORT (28 and Byte'Track (29) on the detection results of DINO and ground truth annotations (test split of object detection), respectively. We report average multi-object tracking accuracy (MOTA),
score (IDF1), false positive (FP), false
negative (FN), and ID switch (IDS) over the videos in testing dataset in Table

---

Table 2: Baselines of action recognition. Average results over three views are reported here and more detailed results can found in the Supplementary document.
Primitive Task
Atomic Action
Method
View
Hant-Hand
Right- Hand
Left Hand
Right
Hand
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
TSM [12
Average
61.0
88.5
58.6
87.9
39.6
69.4
37.0
67.2
Former
Average
52.1
85.4
51.8
84.4
37.6
68.8
34.6
66.1
13D(rgb+flow)
Average
47.7
71.5
52.9
85.1
43.0
75.0
40.5
72.9
MVITv2 [15]
Average
61.5
86.3
48.4
76.5
42.9
71.2
UniFormerV2 [6
Average
62.4
89.7
61.4
48.9
80.9
ST-GCN [17
Average
39.5
60.2
38.7
55.2
20.3
44.4
19.7
40.6
ST-GCN++ 18
Average
38.8
58.0
37.5
56.7
19.0
41.3
16.7
36.1
Table 3: Baselines of action segmentation. Average results over three views reported here and detailed results can found in the Supplementary document.
Primitive task
Atomic Action
Method
View
Left hand
Right hand
Left hand
Right hand
F1
Edit
Acc
F1
Edit
Acc
F1
Edit
Acc
F1
Edit
Acc
MS- TCN 120
Avg.
36.6
37.5
40.2
34.7
34.8
39.3
35.1
32.5
40,9
31.2
32.2
34.6
DTGRM
Avg.
39.1
37.5
37.8
37.3
34.3
32.6
29.8
29.3
BCN (21
Avg.
43.7
41.4
44.1
41.3
38
43,4
18.4
15.9
39.7
22.3
20.1
34.6
C2F-TCN 22
Avg.
22.6
22.0
39.5
22.5
21.9
39
20.5
19.2
37.6
17.4
17.4
31.8
Table 4: Baselines of object detection.
Method
Backhone
Params
AP
AP50
AP75
AP-8
AP-m
AP-I
ResNet50
41.6M
21.7
32.6
24.4
13.0
37.4
40.6
Faster-RCNN 24
ResNet101
60.6M
20.9
31.1
23.9
12.3
37.9
43.1
ResNext101
99.5M
22.2
31.6
25.7
15.0
36.2
46.2
YOLOv5-s YOLOv5-I
DarkNet
7.1M
10.2
14.1
10.9
18.8
DarkNet
46.4M
12.9
14.0
28.8
DINO 26
Swin-l.
218M
35.5
54.5
37.7
27.4
36.4
59.2
Table 5: MOT results object detection results and ground truth object bounding boxes.
Method
bboxes
MOTA
IDFI
FP
FN
IDS
dets
20.4%
27.1%
737.8
9212.3
29
SORT
28
gt
94.5%
69.1%
223.9
408.1
54.8
dets
20.0%
41.1%
5175.3
4678.3
87.2
Bytc Track
29
gt 98.5%
67.5%
32.4
32.5
121.6
The baseline results show that our dataset presents great challenges on the four foundational video understanding tasks compared with other datasets. For example, BCN has 70.4% accuracy on Breakfast [30]. MViTv2 has 86.1% Top-1 accuracy on Kinetics-400 [II, DINO has 63.3% AP on COCO test-dev 32 and ByteTrack has 77.8% MOTA on MOT20 35
Compared to the above baseline results, we are more concerned with whether existing video un- derstanding methods Figure [. Therefore, in Sections 3.2 we further analyze the performance and limitation of the foundational tasks on comprehending application-oriente knowledge, discuss the required assembly video reasoning tasks, and highlight the potential research directions.
effectively comprehend the application-oriented knowledge (shown in
3.2 Assembly progress
Insight #1: Assembly progress understanding could focus on compositional action understand- ing and leveraging prior domain knowledge. Basic assembly progress understanding requires real-time action (action verb + interacted objects and tools) recognition, and compare the action history with the predefined assembly plan (represented in task graph). After further analysis of the sub-optimal action recognition performance Table 2 we found recognizing the interacting objects and tools are more challenging than recognizing the action verbs, (as shown in Table 6).

---

Table 6: Recall of action verb, manipulated object, target object, and tool recognition, via MVITv2.
Action
Manipulated Object
Target Object
Tool
Primitive Task
71.1%
60.4%
60.8% 53.5%
57.1 %%
Atomic Action
67.6%
50.9%
55.0%
Deeper assembly progress understanding requires real-time reasoning on human-object interaction dynamics, future operations, and their operation times. Taking step to address this need, we benchmark FUTR 34 SOTA long-term action anticipation method, and report the mean over classes (MoC) accuracy as per [35 in Table 7 Similar to Table 6 higher accuracy of action verb
anticipation can also be observed in Table 7
Table 7: Mean over classes accuracy of action (action verb interacted objects and tools) anticipation, action verb anticipation and manipulated object anticipation. Following the problem setting in [34] for a video with ''' frames, the first a'l" frames are observed and anticipate the next 3T' frames.
0.2)
(lx=0.3)
0.1
0.2
0.3
0.2
0.3
0.5
Primitive task
7.4
6.1
4.7
4.4
12.3
9.2
6.5
5.6
Action anticipation
Atomic action
2.2
1.8
1.6
1.4
4.7
2.7
2.2
1.8
Primitive task
24.2
23.5
18.2
17.6
32.8
18.6
16.6
16.0
Action verb anticipation
Atomic action
14.6
12.0
10.7
10.4
15.4
13.0
11.7
10.2
Primitive task
13.5
10.0
9.1
9.6
17.0
12.9
10.7
9.1
Manipulated object anticipation
Atomic action
9.0
9.1
8.7
8.1
13.5
10.7
9.8
8.3
Based the observation from Table and 7 promising research direction could be recognizing and anticipating action verbs, objects, and tools compositionally and leveraging prior domain knowledge (such task precedence and probabilistic correlation between action verbs, objects, and tools) to precisely track and predict the assembly progress. With defined task precedence graphs and rich list of annotated action verb/object/tool pairs, HA-ViD enables research on this aspect.
Insight #2: Assembly action segmentation should focus on addressing under-segmentation issues and improving segment- wise sequence accuracy. Assembly progress tracking requires obtaining the accurate number of action segments and their sequence. For obtaining the accurate number of action segments from a given video, previous action segmentation algorithms [19 focused on addressing wer-segmentation issues, but lack metrics for quantifying under/over htte://vve--semmettttionn Therefore, propose segmentation adequacy (SA) to fill this gap. Considering the predicted
segments as Speed = { s2....,'. s'p)
and ground truth segments as 8gt = {81, 82.1,......4 .N)). where
F and N are the number of segments for a given video, SA = tanh (2(F-N) Table reveals the significant under-segmentation issues observed from dataset, which is potentially important issue to be addressed for assembly action understanding, Our proposed SA metric can offer evaluation support, and even assist in designing the loss function given its use of the hyperbolic tangent functionn.
Table Comparison between our dataset and others on segmentation adequacy. We calculated the average number of ground truth segments (N), predicted segments (F), and segment adequacy (S of the videos in the testing datasets of ours and others. The predicted results are from BCN.
Dataset
N
F
SA
Primitive task
14.9
-0.47
HA- ViD(ours)
Atomic action
51.2
11.5
-0.82
Breakfast
-0.12
GTEA
32.5
32.9
-0.03
For segment-wise sequence accuracy, the low Edit value in Table suggests further research efforts required. Compared with Breakfast (30 (66.2% Edit score with BCN), our dataset presents greater challenges.
3.3 Process Efficiency
Understanding process efficiency is essential for real-world industry. It requires video understanding methods to be capable of identifying human pause and error via reasoning of the contextual scene and human-object interaction. HA-ViD supports this research by providing null and wrong labels.
8

---

Insight #3: Null action understanding requires efforts on addressing imbalanced class distribu- tion. Table Sable@ssone shows the recall and precision of action recognition and action segmentation for null actions. We suspect the high recall and low precision is caused by the imbalanced class distribution, as null is the largest head class (see F). To address the class imbalance problem, we randomly sver-sample the minority classes (classes with sample size below a threshold) to reach the threshold, and report the action recognition accurarcy of UniFormerV2 on the over-sampled dataset in Table I0 Here, the threshold is set to 300 and more details can be found in the Supplementary.
Table 9: Recall and precision of null recognition and segmentation. Action recognition results are from MVTTv2 and action segmentation results are from BCN.
Recall
Precision
Primitive Task
90.8%
65.1%
Recognition
Atomic Action
81.5%
39.1%
Primitive Task
80.9
45.1%
Segmentation
Atomic Action
84.6%
37.5%
Table 10: The action recognition accuracy of UniFormeVV2 on the over-sampled dataset.
Primitive Task
Atomic Action
Method
View
Left-hand
Right-hand
Lefi-hand
Right-hand
Top-
Tup-5
Top-1
Top-5
Top-1
Tup-5
Top-1
Top-5
UniFormerV2
Average
62.4
89.7
61.4
89.9
48.9
80.9
44.6
77.6
UniFormerV2 sover-)ampling)
Average
67.3
89.2
66.2
89.1
52.0
81.2
48.8
78.1
Insight #4: New research from wrong action annotations. Wrong actions refer to assembly actions (primitive task level) occurred at the wrong position or order. Our annotated wrong actions can initiate three avenues for research. First, the pattern of wrong actions in different participants across the three-stage progressive assembly can provide insights into human learning progression and performance. Second, investigating the wrong action and the actions leading up to it help identify assembly errors, contributing to quality assurance. Third, analyzing the actions undertaken after the wrong action can provide insights into how humans resolve errors and re-plan the assembly sequence.
3.4 Task Collaboration
Understanding the states, patterns, and dynamics of two-handed collaboration during the assembly process is essential for applications, such as ergonomics analysis, collaborative task planning, and human-robot collaboration design. HA-ViD can support research in this aspect via providing spatial annotations, two-hand separated temporal annotations and collaboration status annotations.
Insight #5: New research on understanding parallel tasks from both hands. Table II shows that both action recognition and segmentation have lowest performance parallel tasks during assembly. One possible reason is that the foundational video understanding methods rely on global features of each image, and do not explicitly detect and track the action of each hand. This calls for new methods to independently track both hands and recognize their actions through local features. Recent research on human-object interaction detection in videos 36 37 could offer valuable insights.
Table 11: Recall of two-handed primitive task recognition and segmentation in four collaboration status. Action recognition results are from MVITv2 and action segmentation results are from BCN.
Action recognition results
Action segmentation results
Collaboration
Parallel
Single-handed
Pause
Collaboration
Parallel Singlc-handed
Pause
Left hand Right hand
52.5%
39.7%
54.2%
92.4% 93.3%
32.1%
15.4%
18.5%
85.5%
46.1%
30.5%
50.7%
35.0%
24.2%
17.2%
82.9%
3.5 Skill Parameters and Human Intention
Understanding skill parameters and human intentions from videos is essential for robot skill learning and human-robot collaboration (HRC)
38 39
Typically, skill parameters vary depending on the specific application. However, there are certain skill parameters that are commonly used, including trajectory, object pose, force, and torque [40] 41

---

While videos cannot capture force and torque directly, our dataset offers spatial annotations that enable tracking the trajectory each object. Additionally, the object pose can be inferred from our dataset via pose estimation methods. Therefore, HA-ViD can support basic research in this direction. To acquire more detailed and accurate skill parameters, assembly process comprehension must extend to the 3D space, utilizing 3D reasoning and 3D hand-object interaction estimation techniques to precisely track the temporal changes in hand poses and objects. For human-robot skill transfer, the learned assembly skill parameters can be sourced as input to robot learning environments.
Understanding human intention in HRC refers to combination of trajectory prediction, action anticipation and task goal understanding [42]. Our spatial annotations provide trajectory informa- tion, A-TGSs present action sequence constraints, and GAB CAD files offer the final task goals. Therefore, HA-ViD ViD can enhance the research in this aspect.
4 Discussion
As identified in Section 3, HA-ViD provides a basis for developing video understanding and rea- soning techniques derive application-oriente knowledge. Model design, results analysis, and knowledge reasoning pipeline development that are based on HA-ViD can accelerate the development of application-specific models. In addition, if a new dataset required for a specific application, our HR-SAT-based annotation protocol can be employed to ensure the resulting target dataset is compatible with HA-ViD. The annotation alignment eases the process of adapting and deploying pre-trained models to new scenarios.
We acknowledge the limitation of HA-ViD on fully capturing the complexities and diversities of industrial assembly scenarios. Therefore, HA-ViD can be extended to include more products, assembly environments, and even different agents. Following our data collection and annotation protocol could ensure similar high-quality assembly video datasets that are compatible with HA-ViD>
Furthermore, we identify that HA-ViD could benefit from 3D and pixel-wise geometric annotations. This can facilitate the research into 3D hand-object interaction and 3D scene understanding, which essential for assembly quality checking and robot skill learning. Therefore, future work could involve providing more refined annotations, such as hand poses, object poses and 3D key points, via additional sensors. We therefore created a dataset roadmap on our Github repository to outline improvement focuses and encourage collective efforts from the community to extend HA-Vin HiD.
5 Conclusion
We present HA-ViD, a human assembly video dataset, to advance comprehensive assembly knowledge understanding toward real-world industrial applications. We designed the Generic Assembly Box to represent industrial assembly scenarios and a three-stage progressive learning setup to capture the natural process of human procedural knowledge acquisition. The dataset annotation follows the Homann-Robo Shared Assembly Taxonomy. HA-ViD includes (1) multi-view, multi-modality data, fine-grained action annotations (subject, action verb, manipulated object, target object, and tool), (2) human pause and error annotations, and (3) collaboration status annotations to enable technological breakthroughs industrial application-oriented knowledge comprehension from videos.
We benchmarked strong bascline methods of action recognition, action segmentation, object detec- tion and multi-object tracking, and analyzed their performance and the further reasoning steps for comprehending application-oriented knowledge in assembly progress, process efficiency, task collab- oration, skill parameter and human intention. The results show that our dataset captures essential challenges for foundational video understanding tasks, and new methods need to be explored for application-orientte knowledge comprehension. We envision HA-Vil will open opportunities for advancing video understanding techniques to enable the futuristic ultra-intelligent industry.
6 Acknowledgements
This work was supported by The University of Auckland FRDF New Staff Research Fund (No. 3720540). Our gratitude extends to the participants and annotators of our dataset for their essential contributions. The Industrial AI Research group has provided invaluable feedback and advice, with
10

---

particular thanks to Benedict Liang for benchmarking assistance, and Saahil Chand for his annotations and insights.
References
[1] D. A. Duque, F. A. Prieto, and J. G. Hoyos, "Trajectory generation for robotic assembly operations using learning by demonstration." " Robotics and Computer Integrated Manufacturing, vol. 57, no. December 2018, pp. 292-302, 2019.
[2] E. Lamon, A. De Franco, L. Peternel, and A. Ajoudani, "A Capability-Aware Role Allocation Approach to Industrial Assembly Tasks," IEEE Robotics and Automation Letters, vol. 4, no. 4, pp. 3378-3385, 2019.
[3] F. Frustaci, S. Perri, G. Cocorullo, and P. Corsonello, "An embedded machine vision system for an in-line quality check of assembly processes," Procedia Manufacturing, vol. 42, pp. 211-218, 2020.
[4] G. Cicirelli, R. Marani, L. Romeo, M. G. Domínguez, J. Heras, A. G. Perri, and T. D' D'Orazio. "The HA4M dataset: Multi-Modal Monitoring of an assembly task for Human Action recognition in Manufacturing," Scientific Data, vol. 9, p. 745, dec 2022.
[5] Y. Ben-Shabat, X. Yu, F. Salch, D. Campbell, C. Rodriguezz-pazo, H. Li, and S. Gould, "The IKEA ASM Dataset: Understanding people assembling furniture through actions, objects and pose," Proceedings 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021, pp. 846-858. 2021.
[6] F. Sener, R. Wang, and A. Yao, "Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities," Cvpr, 2022.
[7] Toyer, A. Cherian, Han, and S. Gould, "Human Pose Forecasting via Deep Markov Models," " DICTA 2017 - 2017 International Conference Digital Image Computing: Techniques and Applications, vol. 2017-Decem, pp. 1-8, 2017.
[8] Zhang, P. Byvshev, and Xiao, "A video dataset of wooden box assembly process: Dataset," DATA 2020 Proceedings the 3rd Workshop on Data Acquisition To Analysis, Part of SenSys 2020, BuildSys 2020, pp. 35-39, 2020.
[9] Ragusa, A. Furnari, S. Livatino, and M. Farinella, "The MECCANO Dataset: Understand- ing Human-Object Interactions from Egocentric Videos in Industrial-like Domain," in 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1568-1577, IEEE, jan 2021.
[10] M. Georgeff and A. Lansky, "Procedural knowledge," Proceedings of the IEEE, vol. 74, no. 10, pp. 1388,1398, 1986.
[11] R. E. Mayer. "Should There Be Three-Strikes Rule Against Pure Discovery Learning?." American Psychologist, vol. 59, no. I, pp. 14-19, 2004.
[12] J. Lin, C. Gan, and S. Han, "TSM: Temporal Shift Module for Efficient Video Understanding." " in 2019 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 7082-7092, IEEE, oct 2019.
[13] G. Bertasius, H. Wang, and L. Torresani, "Is Space-Time Attention All You Need for Video Understanding?." " in Proceedings of the 38th International Conference Machine Learning, pp. 813-824, feb 2021.
[14] Carreira and A. Zisserman, "Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset," in 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4724-4733, IEEE, jul 2017.
[15] Y. Li, C.-Y. Wu, H. Fan, K. Mangalam, B. Xiong, Malik, and C. Feichtenhofer, "MViTv2: Improved Multiscale Vision Transformers for Classification and Detection." in 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1794-4804 IEFE, jun 2022.
11

---

[16] K. Li, Y. Wang. Y. He, Y. Li, Y. Wang, L. Wang, and Y. Qiao, "UniFormerV2: Spatiotemporal Learning by Anming Image ViTs with Video UniFormer," nov 2022.
[17] S. Yan, Y. Xiong, and D. Lin, "Spatial temporal graph convolutional networks for skeleton- based action recognition," 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp. 7444-7452, jan 2018.
[18] H. Duan, Wang, K. Chen, and Lin, "PYSKL: Towards Good Practices for Skeleton Action Recognition," may 2022.
[19] D. Wang, D. Hu, X. Li, and D. Dou, "Temporal Relational Modeling with Self-Supervision for Action Segmentation." " Proceedings of the AAAI Conference Artificial Intelligence. vol. 35, pp. 2729-2737, dec 2021.
[20] Y. A. Farha and J. Gall, "MS-ICN: Multi-Stage Temporal Convolutional Network for Action Segmentation," in 20/9 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2019-June, pp. 3570-3579, IEEE, jun 2019.
[21] Z. Wang, Z. Gao, L.. Wang. Z. Li, and G. Wu, "Boundary- Cascade Networks for Temporal Action Segmentation," ECCV, vol. Part XXV I, pp. 34-51, 2020.
[22] D. Singhania, R. Rahaman, and A. Yao, "C2F-TCN: A Framework for Semi- and Fully- Supervised Temporal Action Segmentation," IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, pp. 1-18, 2023.
[23] Y. Amit and P. Felzenszwalb, "Object Detection," in Computer Vision, pp. 537-542, Boston, MA: Springer US, 2014.
[24] S. Ren, K. He, Girshick, and J. Sun, "Faster R-CNN: Towards Real-Time Object Detec- tion with Region Proposal Networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, pp. 1137-1149, jun 2017.
[25] G. J. C.A. B. WWW.IIKEZZZCAAAAAZZZZOEE U. V. Jain, "YOLOv5,"
[26] H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.-Y. Shum, "DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection," mar 2022.
[27] W. Luo, J. Xing, A. Milan, X. Zhang, W. Liu, and T. K. Kim, "Multiple object tracking: A literature review," " Artificial Intelligence, vol. 293, 103448, apr 2021.
[28] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Uperoft, "Simple online and realtime tracking," in 2016 IEEE International Conference on Image Processing (ICIP), pp. 3464-34444444 3468,, IEEE, sep 2016.
[29] Y. Zhang, P. Sun, Y. Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang, "ByteTrack: Multi-Object Tracking by Associating Every Detection Box," " in Proceedings of the European Conference on Computer Vision (ECCV), vol. 2, oct 2022.
[30] H. Kuehne, A. Arslan, and T. Serre, "The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities," in 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 780-787, IEEE, jun 2014.
[31] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan. F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman, "The Kinetics Human Action Video Dataset," may 2017.
[32] T.-Y. Lin, M. Maire, Belongie, I.. Bourdev, Girshick, J. Hays, Perona, D. Ramanan, C. L. Zitnick, and P. Dollár, "Microsoft COCO: Common Objects in Context," may 2014.
[33] Dendorfer, H. Rezatofighi, A. Milan, J. Shi, D. Cremers, I. Reid, S. Roth, K. Schindler, and L. Leal-Taixé, "MOT20: A benchmark for multi object tracking in crowded scenes," mar 2020.
12

---

[34] D. Gong, J. Lee, M. Kim, J. Ha, and M. Cho, "Future Transformer for Long- temm Action Anticipation." " in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3042-3051, IEEF, jun 2022.
[35] Y. A. Farha, A. Richard, and J. Gall, "When will you do what? Anticipating Temporal Occurrences of Activities," " in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5343-5352, IEEE, jun 2018.
[36] D. Tu, W. Sun, X. Min, G. Zhai, and W. Shen, "Video-based Human-Object Interaction Detection from Tubelet Tokens," in Advances in Neural Information Processing Systems 35, pp. 23345-23357, 2022.
[37] M.-. Chiou, C.-Y. Liao, L.-W. Wang, R. Zimmermann, and Feng, "ST-HOI: Spatial- Temporal Baseline for Human-Object Interaction Detection in Videos," in Proceedings of the 202/ Workshop on Intelligent Cross-Data Analysis and Retrieval, (New York, NY, USA), pp. 9-17, ACM, aug 2021.
[38] O. Mees, M. Merklinger, G. Kalweit, and W. Burgard, "Adversarial Skill Networks: Unsuper- vised Robot Skill Learning from Video," in 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4188-4194, IEEE, may 2020.
[39] P. Zheng, S. Li, L. Xia, L. Wang, and A. Nassehi, "A visual reasoning-basse approach for mutual-cognitive human-robot collaboration," CIRP Annals, vol. 71, no. 1, pp. 377-380, 2022.
[40] J. Jeon, H.-r. Jung, F. Yumbla, T. A. Luong, and H. Moon, "Primitive Action Based Combined Task and Motion Planning for the Service Robot," Frontiers Robotics and AI, vol. 9, feb 2022.
[41] E. Berger, S. Grehl, Vogt, B. Jung, and Amor, "Experience-basee torque estimation for industrial robot," in 2016 IEEE International Conference on Robolics and Automation (ICRA), pp. 144-149, IEEE, may 2016.
[42] Y. Lu, H. Zheng, S. Chand, Xia, Z. Liu, X. Xu, L. Wang, Z. Qin, and J. Bao, "Outlook on human-centris manufacturing towards Industry 5.0," Journal of Manufacturing Systems, vol. 62, pp. 612-611 627, jan 2022.
13


Reviews:
[{"review_id": "OD6JNxqNR7", "reviewer": "Reviewer_HHfr", "summary": "", "questions": "", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper is good and would be useful to community as it contains hierarchical structure, detailed annotations about multi-level task, bboxes around hand, tools, parts, etc. Dataset also covers progressive proficiency in assembly tasks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "FICYzFAK2L", "reviewer": "Reviewer_ei2t", "summary": "", "questions": "", "limitations": "Yes, L233-235.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "HA-ViD is the first one focusing on human assembly videos.  \nIt captures wide-ranging assembly knowledge, including extensive multi-view videos, frames, and labels.  \nThis dataset has potential for research on next-generation intelligent industry applications. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "cFdrHmTe6B", "reviewer": "Reviewer_YGBP", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This work is the first human assembly video dataset for comprehensive understanding of assembly knowledge.\n2. This topic is very interesting and the dataset is completed.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "9XEUr2z9qU", "reviewer": "Reviewer_ZuD2", "summary": "", "questions": "", "limitations": "Yes, the main paper has discussed the limitations.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This dataset is well-motivated. Since assembly knowledge understanding from videos is crucial for futuristic ultra-intelligent industrial applications, the lack of effective data will hinder the development of this field.\n2. This dataset is well designed from the massive captured data, the applicable scenarios, and comprehensive annotations. Compared with previous assembly video datasets, this dataset has a clear superiority. The whole pipeline is sound.\n3. The processes of dataset collection and annotation are clear and informative. \n4. The experiments provide four benchmarks on different tasks. The provided baseline methods are representative, like the action recognition benchmark, using three kinds of methods from five models. In the appendix, most of the experimental details are given.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ffFpGCvtBj", "reviewer": "Reviewer_VsBS", "summary": "", "questions": "", "limitations": "- The task setting: for the assembly task, the action understanding and low-level object task are important, but the assembly instruction reasoning and planning are more vital for a new dataset. There are already many datasets for the provided four tasks. Thus, I suggest the authors redesign the benchmark and propose more reasoning and planning tasks based on the data. This would benefit the finer and more complex action understanding, robot skill learning, human-robot cooperation, etc. For example, next-step prediction, step inpainting, instruction VQA, step order arrangement, etc.\n\n-  The annotation: lacking a very important kind of information for assembly data, i.e., human hand pose. As many actions in the dataset like insert, rotate, etc. At least, estimated and human-adjusted hand 3D key points can be provided. Besides, the Kinetic based 3D pose estimation may be often inaccurate, this should be considered and analyzed.\n\n- Given multi-view videos, readers may want to see more tasks of 3D or 4D reconstructions, hand-object contact detection, etc.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ The assembly understanding matters for many studies, as the authors discussed. A large and well-annotated dataset contributes a lot.\n\n+ Detailed introductions and analyses are given to better understand the data and task.\n\n+ The whole paper is written well and easy to follow.\n\n+ Essential resources are given.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "hSMTnb5IUj", "reviewer": "Reviewer_NUWb", "summary": "", "questions": "", "limitations": "The authors have adequately addressed the limitations.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This article introduces a well-integrated dataset with assembly scenarios, demonstrating significant practical value. It offers four distinct types of annotations that support a variety of video understanding tasks.\"\n\n2. The paper provides a comprehensive showcase of the dataset's data distribution and statistical characteristics, making it easy to understand the dataset's purpose.\n\n3. Through the analysis of experimental results, the authors offer valuable insights into the dataset's potential applications and future directions.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "6qH7EWvFaP", "reviewer": "Reviewer_soUy", "summary": "", "questions": "", "limitations": "They have discussed in short in the conclusion part.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The dataset is challenging and interesting.  \n2. Author has carely considered industrial assembly, and presented some usefull designs.  \n3. The dataset can support multiple tasks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The submission received unanimous support from all reviewers. The AC agrees. The authors are encouraged to revise the submission based on the reviews in the camera-ready version.
Decision: Accept (Poster)