A measure to predict critical errors in open emergency surgery
By: Research Outreach
Tags: Trauma, Asset Procedure, Surgery, Asset Training, Avoiding Surgical Error

Medical errors are a leading cause of worldwide deaths. Surgeons are sometimes required to carry out procedures that they do not perform frequently, which leads to surgical errors that can have severe consequences for patients. To confront this issue, Professor Colin Mackenzie, Emeritus Professor at the University of Maryland, School of Medicine, developed with his team, the Trauma Readiness Index to measure surgeons’ technical skills in open emergency surgery. With this tool, surgeons that are likely to make a critical error can be identified and offered remedial interventions.

An improvised explosive device detonates and the fragments tear a major blood vessel in the lower abdomen. The casualty is bleeding to death and is taken to a local hospital where the surgeon on call makes a mistake of incorrect incision placement, so that the bleeding vessel cannot be controlled before the casualty dies. This scenario is tragic, but similar stories are actually common. Every year in the world, tens of thousands of deaths are related to medical errors.

Medical errors include a wide variety of errors such as inaccurate diagnoses, inappropriate treatments or improper executions of appropriate methods of care. Some have minor repercussions, but others have major impacts. Surgical errors in particular can have severe consequences, including preventable deaths. Most operative adverse events are attributed to technical errors from surgeons.

Initiatives have been developed in an effort to limit the occurrence of surgical errors. An example is the World Health Organization safety checklist, which aims to improve teamwork and communication, skills that are crucial in surgery. However, few studies have focused on errors among individual surgeons, despite their technical skills directly impacting patient outcomes. With this in mind, the American College of Surgeons, Committee on Trauma developed the Advanced Surgical Skills for Exposure in Trauma (ASSET), a one-day course that uses human cadavers to teach surgical procedures known as vascular exposures. Using this course, Professor Mackenzie and his team developed the Trauma Readiness Index to measure the surgeons’ performance.

The ASSET course

To err is human and surgeons are no exception to the rule. The issue is that their errors have more severe and immediate consequences than those of the average physician. A surgical error does not necessarily mean a surgeon is incompetent. Surgical competency involves a combination of abilities such as good decision-making, efficient teamwork and communication (with surgical, anaesthetic, nursing, and other essential staff members), and technical skills. More often than not, errors of various gravity occur during the process, even for the simplest of cases.

Evaluation of ASSET procedure. The head camera captures the procedure and boom microphone allows audio recording of commentary by participant.

One of the reasons why surgical errors occur is the infrequent basis on which specific surgical procedures are performed: when a trauma patient arrives for emergency surgery, surgeons do not have time to go back to their textbooks to refresh their memories; they have to act immediately. This is why repeated training and courses like ASSET can be beneficial for these infrequently performed emergency procedures.

Worldwide, every year, tens of thousands of deaths are related to medical errors.

The ASSET course is open to any surgeon wanting to polish up their anatomical knowledge and technical skills. It uses cadaver surgical simulation to teach exposure of anatomical structures that, when injured, may threaten the associated limb or even life. The course allows surgeons to perform emergency surgical procedures to manage vascular, cardiac and major organ injury that can occur in patients with major trauma.

To assess the effects of the ASSET course, Professor Mackenzie and his team evaluated the performance of participants before and after the course.

Mobile tablet showing app with script, case history and evaluation metrics.

Residents, practicing surgeons and experts

Three different cohorts of surgeons took part in the study. The main cohort was a group of 40 general surgery residents, that is to say qualified physicians, on their way to becoming surgeons. Residents were evaluated a first time before they received ASSET training. They were evaluated a second time within a month after attending the course. Finally, to assess skill retention, they were evaluated a third time about 14 months after ASSET training.

The performance of residents was compared to the performance of 34 practicing surgeons who had received ASSET training between 2 to 4 years prior, and of 10 experts who were surgeons and traumatologists from level 1 trauma centres, hospitals that provide the highest level of surgical care to patients suffering from major trauma injuries.

Four surgical procedures

The technical performance of residents, practicing surgeons and experts was measured during four surgical procedures. Three of them, called vascular exposures, consisted of exposing (uncovering) three different arteries of the arms and of the legs, and in placing a ligature around the vessels before the injury to stop the blood flow. The fourth surgical procedure, called fasciotomy, is a procedure that is needed in case of traumas such as bone fractures or crush injuries which cause compression of nerves and blood vessels: the pressure around blood vessels needs to be relieved to restore blood flow and prevent permanent muscle or nerve damage.

Fig 1: Cluster analysis of correct anatomy skills and technical procedural steps for 4 procedures by 40 trainee surgeon before ASSET training (red); 34 practicing surgeons average 30 months after ASSET (blue) and 10 expert practicing trauma-centre surgeons (green). Cohort centroid is shown by coloured circles. The tertiles of performance are shown for each surgeon cohort as within the coloured circles, above (better performing) the circles and below the circles (lower performing).

Some surgeons in each cohort, including experts, had performed none of these procedures since ASSET training.

Time to complete each of the procedures was limited. The number of critical technical errors was calculated. Indeed, errors were made as surgeons from all cohorts, including experts, incorrectly placed skin incisions, failed to identify a specific artery or failed to relieve the pressure around blood vessels.

Trauma Readiness Index

Surgeons were evaluated and for each of the four surgical procedures obtained an Individual Procedure Score. This score, developed using the same standardised script for all surgeons, reflected both technical and non-technical skills: besides performing the procedures, surgeons were asked questions about diagnosis, patient management, anatomy and procedural steps that also earned points. Technical skills took into account a proper use of instruments, a correct handling of tissues and an overall success of the procedure. Critical errors that were potentially life-threatening were recorded for each procedure. In addition to the number of errors, error recovery was also assessed: surgeons were sometimes able to recognise and correct their errors.

Trauma Readiness Index (TRI) was a global score corresponding to the sum of the Individual Procedure Score obtained for each of the four procedures.

Trauma Readiness Index: Surgeons were evaluated on four surgical procedures and obtained an Individual Procedure Score. This score reflected both technical and non-technical skills.

Benefits of the ASSET course

The ASSET course improves performance: residents obtained a better TRI after receiving ASSET training than before. Moreover, skill retention was good since, 14 months after training, TRI was still as high. Experts had the highest TRI while practicing surgeons had lower TRI than experts and residents after training.

Conducting simple screening tests every two years could limit the number of future critical errors and save lives.

The ASSET course also reduces the number of critical errors: before training, the average number of critical errors was 3.4 per resident but this number went down to 3.0 within a month after training and further down to 1.6 when they were evaluated 14 months later. In comparison, error occurrence was 2.8 in practicing surgeons and 1.4 in experts, which means that, 14 months after training, residents did not make significantly more errors than experts. In addition, before ASSET training, out of 40 residents, none performed the four procedures without making a single critical technical error while 11 residents (27.5% of the group) within a month after training and 8 (21%) of them 14 months later made no critical technical error, something that only 20% of experts achieved.

Finally, the ASSET course also improves the ability to recognise and correct errors. Error recovery is something experts excelled at as they recovered errors 7 times better than residents before training and practicing surgeons and, even though residents had improved at recognising and correcting errors after training, experts still had about 5 times better error recovery.

Fig 2: Cluster analysis of correct anatomy skills and technical procedural steps by same trainee surgeons after ASSET training (red; in comparison to the same practicing surgeons (blue) and expert surgeons (green) shown in Figure 1. Cohort centroid is shown by coloured circles. Note how the red dots of trainee skills now overlap those of practicing surgeons. The upper tertile of trainee surgeons (red dots within green circle) are now performing no differently than experts, the majority of trainees are within the red circle, while 5 of the low performing trainees are shown as the red dots lower than the red circle of their peers suggesting that these trainees showed no benefit from the ASSET course training.

TRI, a useful error predictor

One notable finding was that TRI can predict critical errors: residents with lower TRI before ASSET training were those who made critical errors within a month and 14 months after training. Low TRI was associated with critical errors also in practicing surgeons and experts: a single evaluation using TRI can predict that an individual surgeon will make a critical error that could be limb or life-threatening.

This finding indicates that TRI can be used as a tool to identify surgeons in need of remedial interventions. Professor Mackenzie and his team suggest that conducting simple screening tests every two years could therefore limit the number of future critical errors and save lives.

Personal Response

What sort of remedial interventions would you offer to surgeons who need it? Would you offer them to attend the ASSET course or would there be other possibilities?

The Individual Procedure Scores developed for each of the procedures are based on satisfactory performance of a number of discreet knowledge items and individual steps of a given procedure. Using this assessment tool not only can we identify a surgeon who is not performing at the level of an expert but can also provide individualised feedback as to why. With this information an individualised remediation plan can be developed, tailored to the discrete steps of the procedure and range anywhere from reviewing the anatomy to repeating the individual procedure in a mentored fashion on a cadaver or a simulator, or even repeating the course.

Article References

• Mackenzie C.F., Shackelford S.A., Tisherman S.A., Yang S., Puche A., Elster E., Bowyer M.A., and RASP Investigators. (2019). Critical errors in infrequently performed trauma procedures after training. Surgery, 166, 835–843. https://doi.org/10.1016/j.surg.2019.05.031