As part of a perennial project, our team is actively engaged in developing new synthetic assistant (SA) technologies to assist in training combat medics and medical first responders. It is critical that medical first responders are well trained to deal with emergencies more effectively. This would require real-time monitoring and feedback for each trainee. Therefore, we introduced a voice-based SA to augment thetraining process of medical first responders and enhance their performance in the field.The potential benefits of SAs include a reduction in training costs and enhancedmonitoring mechanisms. Despite the increased usage of voice-based personal assistants(PAs) in day-to-day life, the associated effects are commonly neglected for a study ofhuman factors. Therefore, this paper focuses on performance analysis of the developedvoice-based SA in emergency care provider training for a selected emergency treatment scenario. The research discussed in this paper follows design science in developing proposed technology; at length, we discussed architecture and development and pre-sented working results of voice-based SA. The empirical testing was conducted on twogroups as user studies using statistical analysis tools, one trained with conventionalmethods and the other with the help of SA. The statistical results demonstrated theamplification in training efficacy and performance of medical responders powered byInt J Artif Intell Educ https://doi.org/10.1007/s40593-018-0166-3 *Ahmad Y . Javaid ahmad.javaid@utoledo.edu 1Department of EECS, The University of Toledo, Toledo, OH, USA 2Department of Biomedical, Industrial & Human Factors, Wright State University, Dayton, OH, USA 3Wright State Research Institute, Wright State University, Dayton, OH, USA 4National Center for Medical Readiness, Wright State University, Dayton, OH, USA SA. Furthermore, the paper also discusses the accuracy and time of task execution (t) and concludes with the guidelines for resolving the identified problems. Keywords Emergency care provider .Empirical study .Human performance . Performance analysis .Personal assistant .Synthetic assistant Introduction The emergency care provider (ECP) provides treatment at the point of injury and contributes significantly to the success of a patient ’ss u r v i v a li nc a s eo fe m e r g e n c i e s in both civilian and military domains. Recent studies point out that the number of lostlives due to medical errors is at 251,000 per year, and yet the estimation is considered tobe a low-end number. To this end, medical errors are the third-largest cause of death in the United States, even though they are preventable with increased efforts (Makary and Daniel 2016 ). It is also reported that most common medical errors occur during emergency treatment or transfer of care between different care providers (Hobgoodet al. 2005 ;M o o r m a n 2007 ). Per our estimations, the fatalities caused by medical errors in emergency response account for 50% of the total fatalities. Therefore, to minimizethese errors, a carefully monitored training system for ECPs (for example, a live,virtual, constructive environment) is envisaged to efficiently deal with medical emer-gencies (Henninger et al. 2008 ). Research Background In healthcare, most of the errors take place during information exchange between patients and caretakers due to procedural execution mistakes, inadequate documenta-tion, and poor communication (Nagpal et al. 2012 ). With the application of SA in the medic training phase, it is possible to reduce the number of these errors. Here, the roleof the SA is to monitor the training process in real time and correct the trainee whenever a mistake is committed. The enhanced training module also improves the communica- tion and procedural tactics of medics, thereby reducing the size of the error window (Heet al. 2007 ). Therefore, an SA built with custom features and voice-triggered commu- nication significantly augments a medic ’s capabilities to accomplish a task. Moreover, a voice-based interaction is hypothesized to be better than other modes of communicationin making the treatment process quick, efficient, and user-friendly (Jiang et al. 2015 ; Moorthy and Vu 2014 ). Some of the other technologies, such as a pocket card with a barcode reader, mobile applications, and medical software with touch screens, are also used for information transfer between different medical teams. However, given their inherent drawbacks of being time-consuming and distracting users from primary goals,these interfacing technologies are ineffective and constrained in assisting in complextasks (He et al. 2014 ). Therefore, a real-time, voice-based SA would most likely address all the above-stated drawbacks. A voice query additionally leverages a sophisticatednatural language processing (NLP) Q&A system to produce a natural language re-sponse to the user (Hauswald et al. 2015 ). One of the major problems with the use of speech communication has been substantial recognition errors. With the advent of new cloud-based NLP systems, this problem has been solved to a great extent.Int J Artif Intell Educ Existing studies proved that the replacement of humans with intelligent synthetic agents to train aviation teams would not attribute any loss in training efficiency (Demir et al. 2015 ; Zachary et al. 2001 ). In a generic sense, a synthetic agent or intelligent software agent is defined as "Complex software entities that carry outsome set of operations on behalf of a user with some degree of autonomy" (Franklinand Graesser 1997). SA is a multi-layered intelligent system that sits on top of other services or applications and per forms actions to fulfill the user ’s intent. Here, intent could be a user command to perform an action. SA makes use of a core set oftechnologies, such as machine learning, s peech recognition, dialogue management, language generation, text-to-speech synthesis, data mining, analysis, inference, andpersonalization for performing tasks (Sarikaya 2017 ;Y o r k e - S m i t he ta l . 2009 ). SAs can be useful to a user by providing easy access to personal/external data, webservices, and other applications, such as finding documents, locating places, andplaying music. In addition, they can also provide the user with notifications and alerts, such as alarms, meeting reminders , and meeting locations. These features increase user productivity by assisting them in work and life management (Lee et al. 2015;M i t c h e l le ta l . 1994 ;M y e r se ta l . 2007; Zhang et al. 2016). The evolution of a typical SA started in 1996 and gradually permeated into a software simulationenvironment such as KAoS Reactive Monitoring and Event Notification (developedby The Institute for Human and Machine Cognition, IHMC) (Bunch et al. 2004 ). With the advancement in NLP and image processing algorithms, SA got theupgrade it needed, and this favored easy interfacing with the surrounding environ- ment. SA research further synergized with advanced cognitive concepts to handle high-degree complex tasks (Ball et al. 2010 ). Owing to continuous efforts in cementing perfection within the NLP tech niques, the industry has created several interactive voice-based personal assistants (PAs). Apple Siri, Google Assistant,Amazon Alexa, Microsoft Cortana, and IBM Watson are some of the commerciallyavailable intelligent PAs that provide pro active and reactive assistance to users (Sun et al. 2016). Nowadays, PAs have become an essential capability in most smartphones, tablets, laptops, desktops, and hands-free devices. In general, the user interacts with PAs through natural language commands, and the PA performs an action based on the user ’si n t e n t( S a r i k a y a 2017). These outcomes motivated us to design an interactive voice interface SA for ECPs. Research HypothesisWe hypothesize that a voice-based SA can play a crucial role in the rapid training of ECPs, which can improve their performance and efficiency in the field. To achieve this, we developed an interactive, autonomous voice-based SA to train ECPs. An SA corrects and recommends steps to resolve mistakes committed by a trainee in medicaltreatment and further enables a closely monitored training system to improve ECPsefficiency. A human trainer can observe only one trainee at a time. On the other hand,SAs can monitor multiple trainees simultaneously, designating SAs to be a betteralternative. We hypothesize that the application of SAs renders a higher degree ofmonitoring of each trainee at every phase of training. This helps the human participantto correct the errors during the training phase and has a higher chance of minimizing human errors during execution. The paper is subsequently divided into the followingInt J Artif Intell Educ sections: voice-based synthetic assistant architecture and implementation ,the method- ology of research for user evaluation ,results and discussion ,a n dconclusion . Voice-Based Synthetic Assistant A voice-based SA (VBSA) uses voice interaction as the primary communication mode. Many SA techniques that are task-specific are being implemented by different organi-zations (Ball et al. 2010 ; Zachary et al. 2001 ). Indeed, the developed VBSA is also a task-specific SA. Our subject matter expert, Colonel Douglas C. Hodge, used hisexperience and published work to define the specifications for an ECPtraining-related task. This task was then used to develop the required SA architecture(Moorman 2007 ; Ong et al. 2011; Harden 2011). Based on our subject matter expert specifications, we have identified the following requirements: a) VBSA task: "assist ECP with error detection and recommend steps to resolve errors." b) Human-related task: "provide emergency care to the patient and simultaneously report to SA." c) Interface mode: "do not allow any hindrances to ECP's motor skills while treating the patient." To meet these specifications, the paper classifies (1) process flow monitoring, processed data storage, and training progress statistics under the VBSA task, and (2) patient care. Communication with the SA occurs under the human task. The voiceinterface has become a potential mode of the interface without compromising humangross motor skills in medical treatment. The VBSA is elaborated in two parts, with adescription of design and architecture of VBSA in part I, followed by the realization ofarchitecture with conventional technologies and operation flow in part II. Although thescenario selected for testing the developed VBSA is narrow (only the massive hemor- rhage scenario), the VBSA is developed for the complete BMassive hemorrhage, Airway blockage, Respiration, Circulation, Hypothermia and Eye injury (MARCHE) ^ protocol, also known as MARCHE (Callaway et al. 2011; Savage et al. 2011). Another popular variation of this protocol is MARCH protocol, without the Eye injury compo- nent (Alison 2013 ; Kragh Jr et al. 2009 ). This protocol is used in most combat medic training programs. The applications of VBSA can be used in many parts of medictraining that include training at the point of injury, training for transfer of care, andin-hospital care. It is essential to understand that the list of potential applications is relatively extensive. This technology can be expanded to several other application areas ranging from training in medicine (e.g., monitoring pre-surgery checklist) to simpletraining in education or in a service industry (e.g., assembling parts in an automobile). ArchitectureThe developed SA monitors the training progress of an ECP in real time and helps the ECP resolve errors committed, with predefined measures in addition to the timely alert. The secondary task of the SA is to furnish an ECP with the job information requestedInt J Artif Intell Educ and keep a record of training statistics, including the total number of errors, frequent errors, and the time taken to execute an exercise. To accomplish all these tasks, wereviewed 29 existing autonomous systems such as Coactive Design, ACT-R cognitivearchitecture, and the intelligent-Coalition Formation (Johnson et al. 2014 ). Nine essen- tial functional blocks, including interface, system state control, arbitration, information(processing, collection, and storage), goal recognition and mission planning, dynamic task allocation, rules and roles, validation and verification, and training, were identified throughout the detailed review. As shown in Fig. 1, the architecture comprises a combination of a natural language processor (NLP) and a speech synthesizer (SS) playing the role of the interface and adatabase block. The database block is further comprised of an action sequence, controlsequence, and user profile databases. The primary operation block contains a datapre-processor, post-processor, scenario calculator, process monitor, and result synthe-sizer functions. The database block provides information on mission planning, rules, and roles whereas the result synthesizer serves the functionality of goal recognizer, system state controller, and dynamic task allocator. The communication controller manages communication among functional blocks wherein NLP converts speech information into Text Data (TD). The data pre-processorextracts Scenario Variables (SV = process title names) from the TD received. Further-more, the scenario calculator collects SV and determines the process name to initiate asequence for the process monitor (PM). The primary function of the PM is to monitoreach step of a defined process based on the TD and SV received. In addition, the process monitor creates an error variable (EV) on the notice of errors in a user step and will be passed to a result synthesizer that successively determines the text. The resultsynthesizer sends the text to the data post-processer to construct appropriate sentences.Constructed sentences are then passed onto the speech synthesizer, which in turn relaysthis speech to the user through a headphone or speaker. Table 1represents a comparative analysis of various technologies and developed VBSA to understand their relevance with one another. The technologies mentioned in Fig. 1 V oice-based Synthetic Assistant (SA) architectureInt J Artif Intell Educ Table 1 Comparison of developed VBSA w ith technologies in practice Features Voice interaction level Process mo nitoring capacity Error correction methods Analytics Scalability Integration capability Medic training technology Vo ic e Ba sed Synthetic Assistant(VBSA)Natural language based voice interface withcomplex sentencesSelf-reported step by step process monitorVoice and visual error alert systemTrainee analytics with training history e.g.frequent errorsYes, and accessiblethrough theinternetCan be integrated with prominent medicalsystems Laerdal Robotic ManikinsNo natural language interfaceStep by step process is not monitoredErrors alert is well performed thoughaudible sounds anddataTraining analytics are provided to trainerNo need for physicalhardwareYes, API available made it possible to integrate this. Combat Medic Virtual TrainingNo Voice Process monitor through gamingError reporting through scoringProvides trainee analytics for processYes Data not available Custom PedagogicalAgents in Medical FieldNatural language interface is available in various levels.Feature is available, but an emergency medic training isnot implemented.Visual and voice-based error correctionmethodsTrainee analytics can be obtained.Yes API allows them to integrate with mostavailable software systems.Int J Artif Intell Educ the analysis are selected based on their application to the training of various medical experts. For example, the Laerdal robotic manikin is a commercial product for medicaltraining (Jeffries 2009 ); Combat Medic virtual training is a gaming software for studying paramedic training in military research labs (Graafland et al. 2012 ); and multiple studies of custom pedagogical agents exist in the medical field (Bertrandet al. 2010 ; Spector et al. 2014 ). Most of the robot manikin technologies mentioned here do not have a voice interface and primarily depend on direct touch andhearing-based interaction (hands-on manikin) to simulate a response without sequentialstep monitoring. However, with the integration of sensory interaction and nerve systemsimulation add-ons, the robot manikins can measure trainee implementation withhigher accuracy than the rest of them. On the other hand, game-based virtual simulatorswere considered ineffective by user studies (Barnes et al. 2016 ; Graafland et al. 2012 ) for training due to a lack of physical interaction and perception (hands-on practice andnatural language communication). On the other hand, virtual pedagogical agents are advanced computer-generated avatars that can also monitor a process. On that note, our VBSA can be termed as a pedagogical agent without a computer avatar, effectively ascaled-down version. Although the developed VBSA only interacts via voice, itsarchitectural framework is developer friendly and exhibits compatibility to integratewith other technologies mentioned above, through APIs. For example, VBSA can beintegrated with a robotic manikin, providing manikin sensor data to the data processorin our architecture to recognize task implementation performance through its processmonitoring and database management. Development A cloud platform offers versatility and robustness in addition to profound accessibility in voice-based applications. To date, many cloud service providers provide the toolsrequired including NLP and SS features. Amazon Web Services (AWS), Google Cloud,and IBM Watson are a few examples to notice. With regard to language features andmicro-services available, the developed SA technology is built into AWS. Most of the medical training facilities are equipped with wireless local area network access. This service can be used in most of the training facilities. Still, one of the major problemsusing cloud systems is a loss of connection or communication interruption. In recentyears, the frequency of this problem is reduced significantly. Communication interrup-tions (loss of internet) will make this service unavailable for training. Figure 2shows an SA model utilizing the AWS cloud service to interact with a user. The proposed model uses Alexa V oice Service (A VS) containing NLP, and SS with an Fig. 2 Cloud based SA modelInt J Artif Intell Educ off-shelf smart speaker and mobile application interface called Echo. Although A VS can be trained for different accents without any loss in accuracy, a standard Americanaccent was chosen for the experimental purpose. The database was developed andstored in Amazon Dynamo DB, and primary operation blocks (scenario calculator,process monitor, result synthesizer) are developed in AWS Lambda. The user canaccess the VBSA using the Echo smart speaker or the Amazon Alexa mobile applica- tion. Figure 3shows an actual demonstration of the developed technology where a paramedic is attempting to treat an injured person (manikin with massive hemorrhage on the left leg) with the help of a VBSA using Amazon Echo (shown with a blue ringnear the right bottom corner). The complete demo video can be watched online usingthe link provided in the figure caption. One of the critical questions that may arise with the use of these technologies is storage of medical information in compliance with the Health Insurance Portability andAccountability Act of 1996 (HIPAA). According to the privacy rules of HIPAATitle II, all medical health records of individuals are protected and are to be stored only in a secure location (Terry 2009 ). HIPAA also issued guidelines for cloud computing services to comply with HIPAA regulations by implementing electronic-protectedhealth information (ePHI) by a cloud service provider that allows storing electronichealth records through encryption and password protection on Amazon servers(Schweitzer 2012 ). To ensure HIPAA compliance of our technology, we followed the standard guidelines by Amazon to achieve HIPAA compliance (Amazon Web Services2017 ). Above all, the technology in current usage is aimed at only training and uses fabricated patient health records or information. Working and UsageCurrently, the developed VBSA works on the MARCHE protocol and can be modified to support any training process in emergency medicine. As part of the process, a traineeuses an interface to interact with VBSA, such as an Amazon Echo or an Androidapplication, and our system supports both interfaces. On the other hand, a trainer uses a web-based dashboard, as shown in Fig. 4, to monitor the training progress of all the trainees simultaneously, whereas the administrative staff uses a web-based form as an Fig. 3 Demo: paramedic treating model patient with massive hemorrhage ( https://youtu.be/MNJYfnMta0s )Int J Artif Intell Educ interface to build new training scenarios. The VBSA acknowledges the voice command of the trainee with voice feedback to commence the testing scenario. For example, thetrainee initiates a voice command, such as BAlexa, ask assistant, ^and VBSA acknowl- edges it with either a voice feedback like BOkay ^or simply remains silent. In the background of the conversation, the voice command received by the Echo is convertedinto text through NLP. The data pre-processor analyzes the text to extract keywords, such as Bassistant ^and relays output to the subsequent section. Here, the scenario selector selects a scenariofrom its database, such as Bhemorrhage treatment ^based on keywords extracted in the previous section. As a follow-up, the scenario selector assigns variables to pointexecution steps in hemorrhage treatment and then passes the scenario variables to theprocess monitor. The process monitor tracks the trainee ’s task execution in the treat- ment process and keeps a record of time and errors, such as missing steps. The resultsynthesizer inspects the error variables related to the steps involved in the hemorrhagetreatment and generates a text. Subsequently, the data post-processor constructs text responses that match input from its database and passes the text output to a speech synthesizer. Finally, it is the job of the speech synthesizer to convert the text data into acomprehensive voice response and feedback to the trainee on treatment status. An example conversation between VBSA and a trainee in treating a massive hemorrhage scenario is shown in Fig. 5. In addition, a decision tree diagram approach is depicted in Fig. 6to illustrate a typical VBSA operation. Method The research study was piloted through an experimental setup to scrutinize and validatethe effectiveness of the developed VBSA model as an interactive training companion toassist an individual in executing critical medical tasks. The discussion of the studyapproach is divided into six subsequent sections: (1) Testing scenario ,( 2 )Experimental setup ,( 3 ) Participants ,( 4 ) Procedure ,( 5 ) Data collection , and (6) Data analysis .T h e experiment was conducted on two novice population sets to analyze the performance changes in an individual. The population sets are categorized into two types; a) Fig. 4 Web-based dashboard for trainer to monitor trainees progressInt J Artif Intell Educ individuals trained with conventional methods (called a control set) and b) individuals trained with VBSA (called an experiment set). In the due process, experimentalperformance metric parameters (task execution time, implementation errors, recordingerrors, and subjective cognitive load for the participant; true positive, true negative,false positive, and false negative interactions for VBSA) were collected to learn the effect of the VBSA in training an individual. Tracking performance and error detection ability of the developed VBSA were assessed to learn the relation between theindividual performance and VBSA capabilities. Testing ScenarioThe paper presents a massive hemorrhage treatment scenario selected from the MARCHE protocol of military medic first responder training. An established scenario for the selected activity is as follows: "A patient was wounded badly on his left leg with heavy bleeding and needed immediate medical attention by an ECP." The process is to"stop the bleeding by properly applying two tourniquets on the leg within a givenperiod from the time of task commencement." The task was designed to stop bleedingin hemorrhage treatment, and the selected process was divided into 11 to 13 steps ofimplementation for the participant (acting as an ECP trainee) to follow. The processstarts with the paramedic arriving at the scene to assess hemorrhage and ends with asuccessful application of the second tourniquet to stop bleeding. As a point to note, a simple procedure was chosen for hemorrhage treatment in the experiment. It did not Fig. 5 An example conversation in treating massive hemorrhage scenarioInt J Artif Intell Educ cover an actual process which usually entails a complete list of medical details as well as training material to treat hemorrhage. However, an SA cannot replace a trainer and isonly used to enhance a training program. Experimental Setup The experimental setup comprises a leg manikin model with a massive hemorrhage and an Amazon Echo working as the VBSA desktop interface, as shown in Fig. 7at h a t facilitated the setup of monitoring metrics to oversee the experiment arranged in a quietroom. For the experiment set of participants, the instructions for using Amazon Echowas provided in print with a leg manikin model, as shown in Fig. 7a. For the control set of participants, a leg manikin model and steps for treatment were provided, as shown inFig.7b. The leg manikin used in the experiment was not a complete manikin but was simply developed based on usage. In the pre-experiment briefing, each participant was briefed about the leg manikin setup. The experiment was arranged in a quiet room with Activate SA, e.g., “Alexa Ask Assistant”Point of Injury TrainingTransfer of Care TrainingIn hospital Pre- Surgery Training Airway Respiration Circulation Hypothermia Eye Injuries Pain ControlStep sequence identifierAssessing for Hemorrhage Hemorrhage location and action Applying tourniquet Tightening the tourniquet Reporting bleeding Applying secondary tourniquetV oice I/O Step sequence identifierV oice I/O Step sequence identifier Step sequence identifier Step sequence identifier Step sequence identifierMassive Hemorrhage Briefing Ex: Radio info Evaluation and Observation of Danger Step sequence identifier LEGEND Alexa V oice Service Control/Action Sequence Database User Profile LogEmergency Care Provider (ECP) Training Unidentified InputV oice input/ No InputReminder or unrecognized StatementV oice I/O V oice I/O V oice I/O V oice I/OV oice I/O… Fig. 6 Partial decision tree diagram of VBSA operation for ECP training with hemorrhage scenarioInt J Artif Intell Educ a separate provision for a briefing. Time-stamped video was also recorded with participants ’identity discretion, ensuring that their faces are not visible, as seen from images in Fig. 7. In the experiment, defined performance metrics parameters (task execution time, implementation errors, recording errors, and subjective cognitive loadfor the participant; true positive, true negative, false positive, and false negativeinteractions for VBSA) were recorded independently by two trained observers andVBSA to avoid any human errors. If the recorded values disagreed in any level amongthe observers or with VBSA, then those values were resolved by revisiting therecording of the time-stamped video recorder. Participants A total of thirty-six participants were recruited from the University of Toledo and a local high school. The median of these thirty-six participants ’ages was 22 years (a minimum of 19 years and a maximum of 36 years); seventeen participants were highschool graduates, sixteen participants hold bachelor degrees, and two hold masterdegrees. Among the thirty-six participants in the experiment, fourteen were female.The selection of volunteers was based on English accent to minimize technology bias in the performance of the individual with the VBSA. The test involved reading four different sentences. If the designed VBSA recognized all four at the first attempt, then itwas a 100% score. The participants who scored 100% in English accent proficiencywere taken into the experiment group, and the rest of the participants were assigned tothe control group. This English accent proficiency test is to minimize VBSA bias due toNLP accent limitations. We assigned participants to two population sets, the experimentset containing 19 and the control set containing 17 participants. Our rationale forselecting participants with no medical experience is purely test-case objective, which isBtreating a hemorrhaged right leg (a minor procedure for medics). ^However, it was (a)Experiment set participants training with a VBSA (b) Control set partici pants trainin g without VBSA(c)Testing participants Fig. 7 Experimental setup: participants treating model patient leg with massive hemorrhageInt J Artif Intell Educ understood that recruiting participants with a medical degree or medical knowledge will give a very little or no perspective on studying learning curve of participants. Oneof the critical objectives of this work was to study the utility of VBSA for trainingnovice learners. Therefore, we selected novice participants with very little or nomedical knowledge and briefed them on (i) the treatment process of the selected testcase, and (ii) instructions on how to interact with and use the VBSA at the beginning of their first training round as described in Section 3.4. It was therefore evident that with this methodology, all novice participants will have a similar understanding of hemor-rhage treatment before the experiment, and any pre-knowledge biases will beprevented. ProcedureIt is imperative to develop an experimental procedure with predefined guidelines to ensure the successful implementation of a live scenario. The devised experiment was based on Anderson's 2002 work on studying and understanding human cognition and was grounded in Newell ’s Timescale of Human Activity (Anderson 2002 ,2007 ; Newell 1994 ). The total duration of the experiment was 10 –14 min in varying intervals, as established by Newell ’s, in cognitive performance testing. Moreover, extreme care had been taken to ensure that the participant was a novice and had no prior knowledgeof the testing scenario, even forbidding any contact with a fellow participant in thetraining and testing session. Initially, the control and experimental population sets were briefed individually with the task description narration, followed by a training and testing section for each participant. &Participants from the control population set were trained to treat a massive hemor- rhage, as described in section 3.1 with conventional methods (an instruction sheetbeside the trainee in Fig. 7b), as practiced in conventional training to acclimatize themselves with the task. &Participants from the experimental population set were trained with the help of VBSA (Amazon Echo acting as VBSA and an instruction sheet shown in Fig. 7a) for the same massive hemorrhage treatment. &After the brief training session, both control and experiment participants performed the scenario on a mannequin model leg without any instructions from the examineror the VBSA, as shown in Fig. 7cr e c o r d e da st e s t i n g . Each training and testing session was separated by a considerable time lag to reduce load and information bias among participants. The experimental procedure was iterated thrice, and therefore, supplied three sets of training and testing data. The task partic- ipants trained was to fasten the first tourniquet tightly around the point of injury andcheck the bleeding. Subsequently, the secondary tourniquet was employed to stop themassive hemorrhage completely and bring the patient to a stable condition. The actionperformed by the individual was recorded upon verbal and visual confirmation in eachstep. The critical period set to save the patient was 65 to 95 s, and the ideal time was40 s, as established by a subject matter expert. During the testing phase, participantsfrom both populations received no assistance by any means and were expected to perform alone. The training and testing periods of both population sets wereInt J Artif Intell Educ video-recorded for further evaluation, as per standard data collection guidelines with the test results garnered in a secure system. Data CollectionThe metrics collected from a participant included time of completion, implementation errors, reported errors, and a corresponding number of the error step. Consequently, this enabled mathematical calculation of a performance score, known as the P-score of eachparticipant in the experiment. Indeed, several procedures were implemented to collecteach data point with the least number of errors related to data collection. Two recorderscollected the data simultaneously; also, a timer was implemented in terms oftimestamps to record the data point at each testing and training period. The videorecordings were also gathered for subjective analysis of individual and VBSA perfor-mance. However, the data collected from the experimental population set in training also included a number of interventions by the VBSA, the rate of response, and the tracking performance of the VBSA to the aforementioned data. The SA used in theexperiment cannot distinguish error types corresponding to tourniquet application andonly provided information about time of execution and number of missing stepscommitted by the trainee in a scenario. Furthermore, the VBSA asks the trainee toconfirm bleeding at each step to assess the implementation sequence and decideexecution efficacy. However, the experiment is monitored additionally by two humanobservers in real time recording and reporting errors and execution errors. As a point to note, field errors were reduced by sticking to a strict code of communication and proper lexicon training. Data AnalysisThrough this study, we wanted to determine: (a) if training performance is improved due to VBSA, (b) the effect of VBSA on the speed of task execution, and (c) howthe VBSA affects human performance and accuracy of treatment (e.g., reduction of errors made by VBSA in training). From the recorded metrics, each participant ’s Pscore was calculated based on Eq. 1. The score was assessed by subtracting penalties from a constant number, where each penalty was calculated based on the importance of the task suggested by subject matter experts. The formula is shown below in Table 2and penalty for time provides the possible penalty values that can be generated in this experiment. This formula established in this paper Table 2 Step implementation and reporting error penalty table Step Nos ( i) Step description Penalty for implementation errorPenalty for recording error 1, 2, 7, 9, 11, 12, 13 Checking, assessing, re-assessing, reporting 1 0.5 3, 6 Applying two tourniquets 2 2 4, 5, 8, 10 Tighten tourniquets, check for bleeding 1.5 1Int J Artif Intell Educ helps in statistical analysis that require s generating a unique individual score for every iteration. Pscore ¼10−Penalty for time þ∑13 i¼1Penalty for implementation error þPenalty for recording error0 BB@1 CCA0 BB@1 CCA ð1Þ Penalty for timeif task time t ðÞ<50;penalty ¼0 if50<t<95;penalty ¼t−40ðÞ *0:02 if t>95;penalty ¼10task failureðÞ8 < : The rationale for different steps having different weights are: (1) each steps error leads to a different level of severity, for example, improper application of a tourniquet is more severe than re-assessment step, and (2) implementation error has more weightthan reporting an error in most of the cases. A 10-s adjustment was made as per subjectmatter expert suggestions in the formulation to get the baseline performance score ofnovice participants for this experimentation. An example Pscore calculation in training/testing for experimental/controlled participants is if a participant executed all 13 stepsin 65 s and made errors in the implementation of step 8, then Pscore =1 0−((65− 40) × 0.02 + 1.5) = 8. Data for participants were considered for Pscore calculation only if he or she legitimately performed all experiment steps. Pscore and the total number of errors inboth experimental and controlled groups, for both testing and training cases, arestatistically analyzed to see if VBSA improves the performance. Welch ’st - t e s tw a s used to test the significance of differences in Pscores of the control group and theexperiment group (Ruxton 2006 ). The Wilcoxon Signed-Rank test was used to test the significance of the improvement in performance scores from test 1 to test 3 and from train 1 to train 3 in the same group (Gibbons and Chakraborti 2011); to break the ties, we used the approach defined by (Pratt 1959 ). In a subjective analysis of data, the experimental study also considered the partic- ipants ’feedback over the accessibility of SA in terms of natural language and partic- ipants ’degree of comfort with SA. The examiner observes language exchange and the timing between participant and SA during both training and testing. Even though assessment included positive and negative scale (Watson et al. 1988 ), limitations of this analysis were considered (Naismith et al. 2015 ). Results and Discussion Developed VBSA Results The primary goal of VBSA is real-time monitoring of an ECP trainee with error detection and recommendation of steps to resolve errors. Besides that, SA is alsoInt J Artif Intell Educ designed to record task execution time and errors. Figure 8shows implementation and VBSA operation results, and the video link provided shows the successful operation of VBSA using an experiment test case. At the same time, Fig. 8a is a screenshot of Amazon Dynamo DB database containing MARCHE process steps, Fig. 8b is a snippet of code developed in Amazon Lambda that is responsible for primary blocks operation.Figure 8c is a screenshot of the A VS NLP interface designed for ECP training for the MARCHE process. The interaction effort of VBSA can be represented by observing true positive (TP), true negative (TN), false positive (FP), and false negative (FN) metrics. Every interac-tion or user speech command to VBSA can be categorized in one of those four classesas a stationary value with no probability involved. If a scripted speech command isperformed accurately and VBSA executes it accurately, then it is TP. If a scriptedspeech command is performed accurately, but VBSA rejects it, then it is FN. If ascripted speech command is performed inaccurately and VBSA rejects it, then it is TN. Finally, if a scripted speech command is performed inaccurately, but VBSA accepts any step based on it, then it is FP. We had 19 participants who belonged to the experimentalset that used VBSA for training. Each participant needed to execute 13 steps in each of (a)Treatment process steps developed in DynamoDB Database (b)Sample VBSA Processing code (c) Interface design code for NLP Fig. 8 VBSA development and operational results screenshotsInt J Artif Intell Educ the three training sessions. Figure 9presents the interaction metrics of each of these three sessions represented as Train 1, 2 and 3. The number of total 247 TP interactions for each session resulted from the multiplication of the total number of participants (19)and the required number of steps to be executed for each individual session (13), i.e., 13* 19 = 247. Here, TP and TN represent the correct operation of VBSA interfacewhereas FN and FP represent the error operation of the interface. As we can observe from the graph above, the value of FP is very low and suggests that it is highly unlikely that the VBSA will accept any unwanted command, making the developed systemreliable. User Studies Statistical ResultsAs described in the method section, user studies were performed on the developed VBSA to study the effect of it on the performance of medical trainees. Figure 10 represents the performance score (Pscore) for both the control group and the experi- ment group calculated using Eq. 1for all training and testing iterations. Welch ’st - t e s t found statistically significant improvement in the Pscore of the experiment group in test3 compared to the control group in the same test (z =−2.1538, df = 29.757, p = 0.01974) . As seen in Fig. 10, as the training iterations increase, the mean of the Pscore also increases for both the groups. Welch ’s t-test between the control group and the experiment group on test 2 data determined that the Pscore difference is not statisticallysignificant (p = 0.6395 tested for p < 0.05) . It was also evident that between the two groups, the Pscore difference was not statistically significant (p = 0.2295) for test 1. The two-tailed Wilcoxon sign ranked the differences between tests 1 and 2 (z = −0.6905, p = 0.4902 for p < 0.05) , and between tests 2 & 3 (z =−1.8743, p = 0.06148 for p < 0.05) of the control group showed no statistically significant difference in the Pscore. Nevertheless, statistically significant growth existed from test 1 to test 3 (z = −2.1315, p = 0.03318 for p < 0.05) . The same test applied on the experimental group between tests 1 and 2 (z =−2.1976, p = 0.0278 for p < 0.05) and tests 2 and 3 (z = −3.5162, p = 0.00044 for p < 0.05) indicated a statistically significant difference in 247 247 247 857763 26 28 22 2 3 2 Train 1 Train 2 Train 3VBSA Interaction Metric Chart True Positive True Negative False Negative False Positive Fig. 9 VBSA interface metric chartInt J Artif Intell Educ Pscores, as well as, a statistically significant growth in Pscore from tests 1 to 3 (z = −3.5162, p = 0.00044 for p < 0.05) . In conclusion, Fig. 10illustrates that even after multiple training sessions, the increase in performance (Pscore) was statistically insignificant for the control group, whereas the Pscore of the experimental group has significantly improved from test 1 to test 2 and from test 2 to test 3, and improved significantly compared to control group. Based on the above test results, the hypothesis of the individual performance enhance-ment in training with VBSA when compared to conventional training was mostlyvalidated. Furthermore, we noticed that the repetitive training turned out to be moreeffective with significant growth in the successive tests 1, 2, and 3 when coupled withthe VBSA than in conventional training. These results may change with large samplesand more patient treatment test cases. The total time to complete a given task for the control group and the VBSA assisted group is presented in Fig. 11.W e l c h ’st - t e s ts h o w e dt h a tt h e r ew a sn o Fig. 11 Task time by iteration of train and test with population groups Fig. 10 Pscore by iteration of train and test with population groupsInt J Artif Intell Educ statistically significant loss in execution time between the control group and experiment group in any testing scenarios, which showed that the effect ofcognitive load on individuals due to the VBSA was minimal. As shown in Fig.11and from the analysis performed using two-tailed Wilcoxon sign ranked test (experimental = (z = −2.6694, p = 0.00758 for p < 0.05), control = (z = −2.8682, p = 0.0041 for p < 0.05)) validates our hypothesis that an increase in training iterations reduced the time taken in the testing section by both groups, but thereis no statistically significant differe nce between those two. Based on the above data, it is observed that the training time of the experimental group is longer thanthe time taken by the control group. The discrepancy in training time can beattributed to limited vocabulary interface. Based on the above experiment results and analysis of the time taken to complete a task, the use of VBSA seems to have no adverse effects on the individual ’st a s k execution speed, cognitive load, and stress factor. These results may change with large samples and more patient treatment test cases. The overall distribution of mistakes (All) and the error distributi on of each group is summarized in Table 3. A sr e p r e s e n t e di nT a b l e3 , our tests and observations show a difference between implementation errors (IMP) and reporting errors (REP) in both the control set and the experimental set. Results show that reporting errors are significantly reduced inthe experimental set, which holds well for i mplementation errors also compared to the control group. B a s e do nT a b l e3 results and its analysis, the final question posed about accuracy improvement through VBSA train ing was mostly satisfied by reducing reporting and implementation errors. Res ults indicate that reporting errors were reduced significantly in each training session in the experiment set compared tothe control group. Implementation errors were only reduced from the first set tothe last set of training in the experiment group compared to the control group.Comparing the control set training with that of the experiment set using Fig. 11 and Table 3, we can conclude that implementing a VBSA increases the total training time, which may be related to using scripted language in VBSA, which need further investigation. If further studies with a large sample and more test cases provides any statistical ly significant effects of scripted language in training ECP that effects field performance (testing), this could become a major setback toVBSA that needs to be addressed. Table 3 The two-tailed Wilcoxon sign ranked test result for all PScores in comparison with total number of errors made by individuals in each attempt Train 1 Test 1 Train 2 Test 2 Train 3 Test 3 p <0.05 <0.05 <0.05 <0.05 <0.05 <0.05 Control errors IMP 15 16 8 14 7 9 R E P 2 3 2 71 3 2 21 1 1 1 Experiment errors IMP 9 15 10 15 9 2 R E P 1 6 2 81 0 1 62 5Int J Artif Intell Educ Conclusion It is well known that a large number of deaths result from medical errors. It is therefore obvious that during emergencies, emergency care providers (ECPs) are prone tomaking errors while the accuracy and response time of their actions are critical factorsin the treatment of patients. Any improvement in these factors would significantly increase the chances of survival of a patient. On that note, a primary focus of ECP training should be reducing the response time as well as enhancing the accuracy of ECPresponses. Since conventional methods may not be sufficient in achieving this goal, weproposed and developed a VBSA and evaluated its effect on ECP training. Lessonslearned in developing the proposed VBSA and in testing are as follows: limitations ofVBSA rendered by scripted vocabulary had an observable effect on an ECP ’si n t e r a c - tion with the SA in training that needs further investigation. The possibility ofimplementing a VBSA in real training is viable only if VBSA is subjected to (1) rapid development for real-time application, (2) verification and validation of VBSA, and (3) sustainability of deployment. To conclude, use of an SA for medical training includingseveral aspects of the treatment process is expected to result in efficient and effectivetraining. As the VBSA is capable of handling multiple trainees, it is expected tominimize training costs and enhance utilization of human resources available fortraining without compromising the training quality. The SAs are affordable, upgrad-able, and highly compatible with other existing medical emergency training programsas well as application areas ranging from medicine to industry and education. It can be concluded from the current experiment results that the use of a VBSA in ECP training is a reasonable choice to improve performance and cut down associated training costs.In its current form, the deployment of VBSA is not recommended in training due to itslimitations, such as lack of large sample studies on effects of VBSA on trainees.Rigorous development and more testing are required to address these and otherlimitations that are yet to be discovered. Acknowledgements The University of Toledo and Round 1 Award from the Ohio Federal Research Jobs Commission (OFMJC) through Ohio Federal Research Network (OFRN) fund this research project; authors also appreciate support of the Paul A. Hotmer Family CSTAR (Cybersecurity and Teaming Research) Lab andEECS (Electrical Engineering and Computer Science) Department at the University of Toledo.