# Evaluation of Leadership in Nontechnical Skills

Haruka Ohba, Shinya Mizuno, Yoshikazu Fujisawa, Masashi Uramatsu,
Mikihiro Kano, Naoki Hirabayashi, Takahiro Souma

► To cite this version:

Haruka Ohba, Shinya Mizuno, Yoshikazu Fujisawa, Masashi Uramatsu, Mikihiro Kano, et al. Evaluation of Leadership in Nontechnical Skills. IFIP International Conference on Advances in Production Management Systems (APMS), Aug 2018, Seoul, South Korea. pp.510-517, 10.1007/978-3-319-99704-9_62. hal-02164901

HAL Id: hal-02164901

https://inria.hal.science/hal-02164901

Submitted on 25 Jun 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Evaluation of leadership in nontechnical skills

Haruka Ohba¹, Shinya Mizuno¹[0000-0001-6030-3589], Yoshikazu Fujisawa²,³, Masashi Uramatsu³, Mikihiro Kano⁴, Naoki Hirabayashi⁴ and Takahiro Souma⁵

¹ Shizuoka Institute of Science and Technology, Fukuroi, Shizuoka, Japan

² The Department of Social Engineering and Community Science, Miyagi University, Taiwa, Miyagi, Japan

³ The Department of Patient Safety and Quality, Tokyo Medical University, Shinzuku, Tokyo, Japan

⁴ The Department of Gastrointestinal Surgery, Hiroshima City Asa Citizens Hospital, Asakita, Hiroshima, Japan

⁵ The Department of Medical Safety Management, Chiba University Hospital, Chuou, Chiba, Japan

**Abstract.** Among the knowledge, skills, and abilities related to safety that are required in healthcare practitioners, the present study focuses on skills other than specialized techniques (nontechnical skills), which have drawn much interest, especially in recent years. From our previous research, we know that nontechnical skills may greatly increase the safety of operating rooms, which is also increasingly being recognized by medical workers. Therefore, in this research, we investigated the usefulness of evaluation of nontechnical skills in Japan. We collected data for evaluation at a hospital from between February to and August 2014 using the nontechnical skills (NOTSS) assessment system. The total number of data sets was 270. 18 doctors participated, including five evaluators. We found it difficult for us to evaluate leadership in NOTSS. To perform NOTSS in Japan, different evaluation indicators may be necessary that change to leadership. We will propose that indicator in the future.

**Keywords:** Nontechnical skill · Leadership · ANOVA.

## 1 Introduction

Among the knowledge, skills, and abilities related to safety that are required in health-care practitioners, the present study focuses on skills other than specialized techniques (nontechnical skills), which have drawn much interest, especially in recent years. Research concerning human error in aviation started in the 1960s because flight crew were unable to say anything to the aircraft pilots, who wielded absolute power, which led to aircraft accidents[1][2]. Previous studies[3][4][5] have shown that nontechnical skills are essential to increase safety standards further, which is increasingly being recognized by health-care practitioners. The construction of an effective and efficient evaluation system for nontechnical skills is extremely important, and the development of educational
programs to link the results of such an evaluation system with the improvement of nontechnical skills is essential[6]. Nontechnical skills can be examined in all facets of medical care, however, because surgical treatments are invasive and put patients at high risk, increasing safety is especially in line with policy priorities. Therefore, the present study focuses on the field of surgery[7]. The principal aim of the present study is to evaluate of leadership in nontechnical skill for surgeons (NOTSS). It is often said that Japanese do not demonstrate leadership, so we confirm it in actual surgical operation. In this study, it is an analysis of leadership using medical data, but we think that this study has needs in a wide field including such as management engineering.

## 1.1 The Importance of Nontechnical Skills

It has become clear that when performing medical activities, most problems that occur during operative treatment are related to nontechnical skills (Table 1). It has been reported that 43% of surgical errors are due to a lack of communication[8] and 97% of bile duct injuries are due to a failure to recognize a situation[9].

<table><caption>Table 1. Report on medical accidents. (Japan Council for Quality Health Care[10]).</caption><thead><tr><td>Reason for accident</td><td>Percentage</td></tr></thead><tbody><tr><td>Knowledge was insufficient</td><td>3.4%</td></tr><tr><td>Technical skill was insufficient</td><td>2.4%</td></tr><tr><td>Confirmation was neglected, observation was neglected, or explanation was insufficient</td><td>59.2%</td></tr><tr><td>others</td><td>35%</td></tr></tbody></table>

Furthermore, *the World Health Organization (WHO)* has published a standardized guide for hand hygiene and a safety checklist for operative patients[11], and a global movement to improve patient safety has begun. *The WHO* safety checklist for operative treatment has been tested in various countries worldwide and the incidence of complications and mortality associated with operative treatment decreased 36% and 47%, respectively, after the introduction of the checklist.

There has been a trend to focus on skills related to medical treatment to achieve good results, but whether or not the medical practitioners possess superior skills, the team performance is low when individual situation awareness and decision-making skills are insufficient, when communication with other team members cannot be performed well, or when leadership cannot be exercised. Therefore, we introduce an educational program for the specialization of nontechnical skills for surgeons during operative treatment to ensure the maximization of the operative team performance[12].
## 1.2 Organization of nontechnical skills

To utilize the NOTSS system and assess behaviors effectively, Yule, S., Rowley, D., Flin, R., et al.,[13] noted that the following three points are necessary: (i) fundamental knowledge concerning NOTSS and human performance and error management, (ii) fundamental principles in the use of psychometric tools to understand the NOTSS system and to evaluate behaviors in the clinical setting, and (iii) training in the use of a corrective program to construct objective evaluation criteria. To achieve these points, Yule, S, et al.,[13] categorized nontechnical skills into four groups, and each skill was further subcategorized into three elements (Table 2).

<table><caption>Table 2. NOTSS: four skill classifications and the three elements for each classification.</caption><thead><tr><th>Classifications</th><th>Elements</th></tr></thead><tbody><tr><td>Situation awareness</td><td>Gather information<br>Understand information<br>Foresee the future and act accordingly</td></tr><tr><td>Decision making</td><td>Examine choices<br>Make a choice and share with team members<br>Carry out the choice and confirm the progress</td></tr><tr><td>Communication and teamwork</td><td>Share information with team members<br>Build mutual understanding<br>Coordinate team behaviors</td></tr><tr><td>Leadership</td><td>Set performance standards and maintain them<br>Support members<br>Manage team pressure</td></tr></tbody></table>

Using evaluation criteria as such as that shown in Table 3, each category was scored and the evaluation criteria were standardized.

<table><caption>Table 3. NOTSS: standardization of the evaluation criteria.</caption><thead><tr><th>Criteria value</th><th>Evaluation criteria contents</th></tr></thead><tbody><tr><td>4. Good</td><td>The operative treatment performance was consistently high. Standards were maintained and treatment was also a good example for the promotion of patient safety</td></tr><tr><td>3. Acceptable</td><td>Standard and satisfactory performance, but with room for improvement</td></tr><tr><td>2. Marginal</td><td>Elements for concern were present and there was considerable room for improvement</td></tr><tr><td>1. Poor</td><td>Patient safety was compromised and there were potentially dangerous elements present. Significant improvement is needed</td></tr><tr><td>N/A. Not applicable</td><td>In this case, “skill” was not necessary</td></tr></tbody></table>
# 2 Evaluation of nontechnical skill

## 2.1 Basic nontechnical skill evaluation data.

Evaluations were performed on site using the NOTSS assessment system. We collected data at a hospital from between February to and August 2014. The medical facility had 527 beds and 150 doctors. Nineteen surgeons, including six residents, work in the 60-bed surgical ward. About 1200 cases are operated annually. We assume a target performance of one operation per day. We included surgeons who have performed the same procedure more than 10 times. Therefore, high-risk operations were not included and infrequent procedures in which surgeons do not have much experience were not included. We also assumed that emergency surgeries would be excluded from this study because the NOTSS assessment system should be used in operations when the surgeon has plenty of time to prepare for the operation and can usually operate without being stressed during the procedure.

Evaluation was performed by having a single evaluator evaluate a single surgeon. The total number of data sets was 270. Eighteen doctors participated, including five evaluators. We removed an executive surgeon from the operator evaluations for the statistical analysis. The evaluation data was input by each evaluator using a personal computer or tablet. In the data tabulation, each element was scored out of a total of 48 points, with three elements in each category. Each element had a maximum score of four points. Therefore, each category had a total of 12 points, with 48 points in total. Data for elements that did not have an evaluation value were treated as missing values as appropriate. The NOTSS evaluation data is shown in Table 4. From these results, the evaluation of situation awareness has high scores and small variations, but the leadership evaluation shows that the scores vary with low scores. For the correlation coefficient, the value related to leadership is low.

<table><caption>Table 4. Basic NOTSS evaluation data (Mean, standard deviation, correlation coefficient).</caption><thead><tr><th>Classifications</th><th>Mean</th><th>Dev</th><th>Situa</th><th>Decision</th><th>Commu</th><th>Leader</th></tr></thead><tbody><tr><td>Situation awareness</td><td>10.03</td><td>3.83</td><td>1.0</td><td>0.87</td><td>0.81</td><td>0.67</td></tr><tr><td>Decision making</td><td>9.94</td><td>4.39</td><td>0.87</td><td>1.0</td><td>0.86</td><td>0.73</td></tr><tr><td>Communication and teamwork</td><td>9.89</td><td>4.53</td><td>0.81</td><td>0.86</td><td>1.0</td><td>0.72</td></tr><tr><td>Leadership</td><td>9.49</td><td>5.66</td><td>0.67</td><td>0.86</td><td>0.72</td><td>1.0</td></tr></tbody></table>

Next, when performing the principal component analysis, the results are as shown in the Fig 1. From Table 5, principal component 2 has a positive leadership value, and the influence of leadership is significant. From Fig 1, we get that the variation in the PC 1 direction is small, but the variation in the PC 2 direction is large. From this result, we think that the quality of evaluation differs from that of other categories in the evaluation of leadership.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Principal component analysis of 4 categories.</figcaption></figure>

<table><caption>Table 5. Principal component analysis of 4 categories.</caption><thead><tr><th>Classifications</th><th>PC1</th><th>PC2</th><th>PC3</th><th>PC4</th></tr></thead><tbody><tr><td>Situation awareness</td><td>0.504</td><td>-0.424</td><td>0.563</td><td>0.498</td></tr><tr><td>Decision making</td><td>0.520</td><td>-0.240</td><td>0.075</td><td>-0.815</td></tr><tr><td>Communication and teamwork</td><td>0.509</td><td>-0.122</td><td>-0.801</td><td>0.286</td></tr><tr><td>Leadership</td><td>0.464</td><td>0.864</td><td>0.184</td><td>0.058</td></tr><tr><td>Cumulative Proportion</td><td>0.833</td><td>0.925</td><td>0.971</td><td>1.000</td></tr></tbody></table>

## 2.2 Analysis by two-way factorial ANOVA without replication.

We are interested in whether NOTSS was different for every surgeon and every category. Using a statistical method, we analyze the obtained data to derive our results. We categorize two types, such as surgeon and evaluator, and analyze by two-way factorial analyses of variance (ANOVA) without replication.

We focus on the mean for each surgeon. Fig 2 indicates four categories of means for each surgeon. We used two-way factorial ANOVA without replication for Fig 2 to produce an ANOVA table (Table 6). From Table 6, we find $P = 4.84E-23$ for surgeons. Therefore, significant difference is found between surgeons for a mean NOTSS with 95% significance. Similarly, we find $P = 1.87E-4$ for categories. Thus, there are significant differences between categories.

## 2.3 ANOVA of categories

We know there are significant differences in means and variance for surgeons (Table 6). Here, we explore which pairs of categories have significant differences.

As shown in Fig 2, the horizontal axis indicates the Surgeon ID. Fig 3 indicates the boxplot for four categories. We can find the whole to have a low item
<figure><img src="image_3.png" /><figcaption>Fig. 2. Four categories of means for each surgeon.</figcaption></figure>

<table><caption>Table 6. ANOVA table of four categories of means for each surgeon.</caption><thead><tr><th>Variation factor</th><th>Variation</th><th>Degree of freedom</th><th>Variance</th><th>Observed variance ratio</th><th>P-value</th><th>F boundary value</th></tr></thead><tbody><tr><td>Surgeon</td><td>124.203</td><td>12</td><td>10.350</td><td>89.163</td><td>4.843E-23</td><td>2.033</td></tr><tr><td>Category</td><td>3.012</td><td>3</td><td>1.004</td><td>8.650</td><td>1.870E-4</td><td>2.866</td></tr><tr><td>Error</td><td>4.179</td><td>36</td><td>0.116</td><td></td><td></td><td></td></tr></tbody></table>

of the leadership. First, we use the Bartlett test to confirm the homogeneity of variances (P = 0.116) for these categories. We set the significance level at 5%. The null hypothesis is that the mean of the population of each category is equal. An alternative hypothesis is that there is a difference in the means in at least one set in each category. We use the Bonferroni adjustment method to obtain P-values (Table 7). Thus, we find that leadership differs from the awareness, decision, and communication categories with 5% significance.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Boxplot for the surgeon means in each category.</figcaption></figure>

## 2.4 Analysis using surgeon's experience and Surgical difficulty

Next we analyzed using surgeon's years of experience and surgical difficulty level [14]. Surgical difficulty is classified in three level. The correlation coefficient be-
<table><caption>Table 7. ANOVA results for means of categories.</caption><thead><tr><th>Pairs</th><th>Awareness</th><th>Decision</th><th>Communication</th><th>Leadership</th></tr></thead><tbody><tr><th>Awareness</th><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><th>Decision</th><td>1</td><td>-</td><td>-</td><td>-</td></tr><tr><th>Communication</th><td>1</td><td>1</td><td>-</td><td>-</td></tr><tr><th>Leadership</th><td>1.00E-03</td><td>2.27E-02</td><td>5.40E-03</td><td>-</td></tr></tbody></table>

tween experience years and surgical difficulty level is 0.70, which shows that a veteran doctor is conducting a surgery with a high degree of difficulty. From the Fig 4, the higher the surgical difficulty level, the higher the leadership evaluation tends to be. However, there are similar trends in other categories, not limited to leadership. Scatter plot of surgical difficulty level and leadership evaluation is shown in Fig 5. The correlation coefficient between experience years and leadership evaluation is 0.64, which is the largest among the categories. The variance of the leadership evaluation of doctors with experience years less than 10 years is 5.09, which is the largest among the categories. On the other hand, the distribution of the leadership of doctors with more than 10 years of experience has fallen sharply to 1.34. From this data, we consider that the years of experience has an influence on leadership.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Histogram of surgical difficulty level and leadership evaluation. Fig. 5. Scatter plot of surgical difficulty level and leadership evaluation.</figcaption></figure>

## 3 Conclusion

In this study, it is difficult for us to evaluate leadership in NOTSS. There are differences in values for the respective roles and we need to standardize this aspect of evaluation. Especially for Japanese, leadership is difficult to evaluate. To perform NOTSS in Japan, different evaluation indicators may be necessary that change to leadership. We will propose that indicator in the future. These study results show that the experience of the surgeon is influential for NOTSS. Therefore, hospitals should necessarily cooperate with an education system to
provide NOTSS to their staff [15]. We hope that NOTSS will be applied widely
in hospitals.

References

1. Helmreich, Robert L., and José M. Anca. Crew resource management. Ed. Barbara G. Kanki. London: Elsevier (2010)

2. ANDRIES, Cristian. CREW RESOURCE MANAGEMENT. Defense Resources Management in the 21st Century (2013)

3. Soma, T. Development of an educational training program for nontechnical skills in the field of surgery and the construction of an evaluation system for the program, Annual Report for 2012 (2012)

4. Soma, T: Nontechnical Skills from the Viewpoint of Medical Safety-Specific Problematic Behaviors as Seen from Surgeon Training Programs in Australia and New Zealand, Clinical Surgery 68(7)764-772 (2013)

5. Eefje N. de Vries, Hubert A. Prins, Rogier M.P.H. Crolla, Adriaan J. den Outer, George van Andel, Sven H. van Helden, Wolfgang S. Schlack, M. Agns van Putten, Dirk J. Gouma, Marcel G.W. Dijkgraaf, Susanne M. Smorenburg, and Marja A. Boermeester: "Effect of a comprehensive surgical safety system on patient outcomes." N Engl J Med. 363:pp.1928-1937 (2010)

6. Rhona Flin and Steven Yule, Report on the University of Aberdeen's Non-Technical Skills for Surgeons project, Advances in patient safety: non-technical skills in surgery Surgeonsnews 4: pp.83-85 (2005)

7. Vincent C, Moorthy K, Sarker SK, Chang A, Darzi AW.: "Systems Approaches to Surgical Quality and Safety From Concept to Measurement." Annals of Surgery, 239: pp.475-482 (2004)

8. Gawande AA, Zinner MJ, Studdert DM, Brennan TA : "Analysis of errors reported by surgeons at three teaching hospitals. " Surgery. 133:pp.614-621(2003)

9. Lawrence W. Way, Lygia Stewart, Walter Gantert, Kingsway Liu, Crystine M. Lee, Karen Whang, and John G. Hunter : " Causes and prevention of laparoscopic bile duct injuries: analysis of 252 cases from a human factors and cognitive psychology perspective " Ann Surg. 237(4): pp.460-469 (2003)

10. Japan Council for Quality Health Care : The 20th report (2009)

11. World Health Organization: Multi-professional Patient Safety Curriculum Guide (2011)

12. R. Flin, R. Patey, R. Glavin and N. Maran.: "Anaesthetists' non-technical skills, British Journal of Anaesthesia 105 (1), p38-44 (2010)

13. Yule, S., Rowley, D., Flin, R., Maran, NR., Youngson, G.G., Duncan, J., Paterson-Brown, S.,Experience matters: Comparing novice and expert ratings of non-technical skills using the NOTSS system, ANZ Journal of Surgery 79, pp.154-160 (2009)

14. The Japanese Society of Gastroenterological Surgery , Gastroenterological surgeon specialist training curriculum, https://www.jsgs.or.jp/modules/others/index.php?content_id=7 Last accessed 30 Mar 2018

15. Yule, S., Flin, R., Rowley, D., Mitchell, A., Youngson, G., Maran, N. and Paterson-Brown, S. : "Debriefing surgical trainees on non-technical skills (NOTSS). Cognition. " Technology & Work, 10, pp.265-274 (2008)