# Instrument Detection and Descriptive Gesture Segmentation on a Robotic Surgical Maneuvers Dataset
## 2 Materials and Methods
### Gesture Annotations
#### 2.3.3 Fine-Grain Descriptor (FGD) Gestures  
FGD gestures describe the gestures at lower level, i.e., the adjectives used to define the gesture are linked to specific actions of the tools. We have defined six FGD gestures, which are common for the two tasks of the dataset. Most of them can be seen as a particularity of a MD gesture; the relationship between them is shown in Table 5. Additionally, Table 6 presents the ID, label and description of each gesture, along with the number of frames annotated for PSM1 and PSM2. These six FGD gestures are as follows:  
Figure 4: Examples of MD gesture annotations. Figure 5: Distribution of the gestures occurrence for the MD gestures for (**a**) PSM1 and (**b**) PSM2. The red central mark on each box indicates the median, and bottom and top edges indicate the 25 th and the 75 th percentiles, respectively. The whiskers extend to the most extreme data points not considered outliers, and the outliers are plotted individually using the ‘+’ marker symbol. * _Idle (F1)_: this is the same gesture as for MD gestures described in the previous section (_G1_). * _Picking (F2)_: the instrument is picking an object, either a pea on the _pea on a peg_ task or a colored sleeve on the _post and sleeve_ task. This gesture is a particularity of _G2_ of MD descriptors. * _Placing (F3)_: the instrument is placing an object, either a pea on top of a peg or a sleeve over a peg. This gesture is also a particularity of _G2_. * _Free motion (F4)_: the instrument is moving without carrying anything at the tip. This gesture corresponds with actions of approaching the objective to pick, and it is a particularity of _G3_ of MD descriptors. * _Load motion (F5)_: the instrument moves while holding an object. This gesture corresponds with actions of approaching the objective to a place, and, therefore, it is also a particularity of _G3_. * _Collaboration (F6)_: equivalent to gesture _G4_ of maneuver descriptors. Figure 6: Sequential distribution of the MD gestures along a complete trial of a pea on a peg task for PSM1 (**left**) and PSM2 (**right**). Figure 7: Sequential distribution of the MD gestures along a complete trial of post and sleeve task for PSM1 (**left**) and PSM2 (**right**). Figure 8 shows six characteristic snapshots of each FGD gesture, three for _pea on a peg_ (top) and three for _post and sleeve_ (bottom images). Figure 9 shows the gesture occurrence distribution for FGD gestures. As expected, this box chart reveals the same conclusions of MD gestures in term of the dispersion of the data for the different task trials. This can also be seen in Figures 10 and 11, which show the sequential workflow of the gestures for a pea on a peg task and a post and sleeve task, respectively:  
* The workflow of _pea on a peg_ task for FGD gesture is mainly as follows: _free motion (F4)-picking (F2)-load motion (F5)-placing (F3)_, with short interruptions for _collaboration (F6)_. The other tool is mostly in an _idle (F1)_ position, with some interruptions for _collaboration (F6)_. * As we stated previously, _post and sleeve_ tasks follow a more sequential workflow between the tools. In the trial represented in Figure 11, PSM1 was the dominant tool, so the comparison between the gesture sequential distribution for PSM1 and PSM2 reflects that _picking_ is a more time-consuming task than _placing_.