Testing NOTSSdk in the Clinical Setting
Condition: Focus of Study: Usability of a Tool for Structured Assessment of and Feedback to Trainee Surgeons
Study type: Observational

Summary: 
      The purpose of this study is to test the use of a tool (NOTSSdk)to assess and provide
      feedback to trainee surgeons' non-technical skills. It focuses on two aspects: reliability of
      ratings of trainee surgeons using NOTSSdk; and the content and quality of feedback given to
      trainees based on observations using NOTSSdk.
    

Detailed description: 
      The specific research questions are:

        -  How many situations (=operations) are necessary to assess using NOTSSdk, to get reliable
           ratings of trainee surgeons' NTS?

        -  What is the impact of the NOTSSdk feedback tool on the content of feedback conversations
           regarding NTS?

        -  How do trainees perceive the usefulness of the feedback session?

        -  How do raters perceive the usefulness of the ratings and the feedback session?

      Materials and methods

      We aim for a sample of 15 trainees in one hospital.

      Demographic data will be collected regarding: gender, age, postgraduate experience
      (years/months), surgical experience (years/months), specialist training level for both
      assessors and trainees.

      Procedure The head of the surgical department will be informed about the project and will be
      asked permission to recruit trainees and assessors.

      Participants will be informed about the project orally and in writing prior to commencement.
      Surgeon assessors and trainees will be given the NOTSSdk handbook to familiarize themselves
      with the system before the study.

      The assessors will receive a short guidance on how to provide assessments and give feedback
      using NOTSSdk immediately before the first operation. After the first assessment and feedback
      session, the assessor will be offered a further short structured guidance on how to fill in
      the assessment form and how to provide structured feedback. It will also be allowed for the
      assessor to solve ambiguities, ask questions and to get further explanation of the categories
      and elements of NOTSSdk.

      Eligible operations: laparoscopic cholecystectomy, hernia repair, laparoscopic appendectomy,
      small bowel obstruction, hemicolectomy - depending on the trainee's competence.

      Reliability study Each trainee will be assessed by one surgeon assessor using NOTSSdk during
      an operation. Each trainee will be assessed on two occasions. Assessors will vary according
      to availability. This will produce app. 30 assessments in total.

      Feedback study Data will be collected in connection with data gathering in the reliability
      study that is, data will be provided by the same assessors and the same trainees on the same
      occasions. After the operation the surgeon will provide feedback to the trainee based on
      his/her observations. This feedback session will be observed and audio recorded by the
      primary investigator. After each feedback session the trainee and the surgeon will be
      individually surveyed exploring their perception of the feedback session and its usefulness.
      After the last feedback session the assessors will be surveyed about their perceptions of
      NOTSSdk feasibility.

      Data analysis

      Reliability study Generalizability theory will be applied to analyze how many operations are
      necessary to observe for reliable assessment. First the different variance components
      (trainees, assessors, cases etc.) will be estimated (three facet, balanced design with
      trainees nested within assessors and nested within operations). These will be used to
      generate Generalizability coefficients to estimate overall reliability of the use of NOTSSdk
      depending on number of assessors and operations observed.

      Feedback study

      We will use qualitative content analysis of emergent themes to explore the contents of the
      feedback sessions with special focus on:

        -  Frequency of NTS topics discussed over time

        -  Frequency of TS topics discussed over time Descriptive statistics will be used to
           present trainees' perceptions of the usefulness of the feedback and to present
           assessors' perceptions of the usefulness and feasibility of the ratings and the
           feedback.
    

Observational model: Cohort
Time perspective: Prospective
Sampling method: Non-Probability Sample
Study population: 
        Trainee surgeons in one surgical department