Reliability and Validity of a Tool Measuring Preceptor Evaluations of Competencies among New RN Graduates in a Transition-To-Practice Program

Four transition-to-practice programs for new registered nurse (RN) graduates who had not yet found employment in nursing were based on Quality and Safety Education for Nurses (QSEN) competency knowledge, skills, and attitudes (KSAs). To support consistent evaluation of participants by preceptors, a 35-item tool was developed that uses a three-point scale to assess selected behaviors across all six QSEN competencies. The purpose of this report is to describe initial reliability and validity testing of the tool. It has good internal consistency with a Cronbach’s alpha of 0.92. Six content experts in nursing education positively evaluated the face validity of the tool, and it discriminated between junior-level baccalaureate nursing students and nursing faculty who self-rated their competencies. Although it does not exhaustively reflect the QSEN KSAs, the tool requires approximately ten minutes for completion, providing a practical way to assess development of QSEN competencies among new graduates. RN Transition Program Competency Assessment Tool 3 For newly graduated registered nurses (RNs), the transition from student to advanced beginner professional nurse is challenging. A pivotal 2010 Institute of Medicine (IOM) report on the future of nursing identified post-graduate residencies as one of eight recommendations for preparing nurses for effective professional practice (Institute of Medicine, 2010), and evidence is accumulating about the effectiveness of nursing residencies. In a 2012 review of 20 studies evaluating residency programs for new RN graduates, organizational outcomes, such as recruitment, retention, and turnover, were most frequently measured (Anderson, Hair, & Todero, 2012). The most frequently used tool employed in nursing residency evaluations was the CaseyFink Graduate Nurse Experience Survey measuring self-perceived competence (Casey, Fink, Krugman, & Propst, 2004). Performance-based clinical competence was assessed by only three studies (Beecroft, Kunzman, & Krozek, 2001; Cleary, Matheson, & Happell, 2009; Herdrich & Lindsay, 2006), using instruments measuring professional autonomy (Schutzenhofer, 1987), nursing competencies (Wandelt & Stewart, 1975), and performance of nursing behaviors (Schwirian, 1978). Based on the IOM competencies for nursing (Institute of Medicine, 2003), the Quality and Safety Education for Nurses (QSEN) initiative defined six safety and quality competencies and proposed knowledge, skills, and attitudes to be developed for each: patient-centered care, teamwork, evidence-based practice, quality improvement, safety, and informatics (Cronenwett et al., 2007). QSEN competencies are being broadly integrated into prelicensure curricula and are assuming increasing importance among inpatient nurse leaders (Barnsteiner et al., 2013; Berman, Johnson, & West, In press; Falls & Hensel, 2012). However, to the best of our knowledge, no psychometrically validated measurement tool exists for assessing the performance of new RN graduates related to QSEN competencies. The purpose of this report is to describe the RN Transition Program Competency Assessment Tool 4 development and validation of a tool with which clinical preceptors can assess the performance of new RN graduates with respect to QSEN competencies. Tool development In 2010, four academic-practice partnerships in California offered transition-to-practice programs for new RN graduates who had not yet found employment in nursing (Berman et al., 2014; Jones-Bell et al., 2014; Wallace et al., 2014; West et al., 2014). Each partnership had the flexibility to design a program to best meet the needs of the clinical practice sites but all developed curricula based on the QSEN competencies (West et al., 2014). A tool congruent with the QSEN framework was needed to evaluate the effect of the program on participants’ performance. Clinical nursing leaders from three facilities collaborated to develop a tool to evaluate the performance of new graduate participants in terms of QSEN competencies. Over a period of several weeks, the workgroup translated QSEN knowledge, skills, and attitudes (KSAs) into discrete observable behaviors, adapting and expanding on a recently published assessment of new RN graduate proficiencies (Berkow, Virkstis, Stewart, & Conway, 2009). The tool was subsequently approved by the transition program’s evaluation advisory committee. Table 1 displays the behaviors related to QSEN competencies included in the tool; the 36th item asks preceptors to evaluate the overall clinical competency of the new RN graduate. A four-point scale indicated the level of competency: 0 = not applicable, 1 = beginning competency, 2 = developing competency, and 3 = advanced competency. RN Transition Program Competency Assessment Tool 5 Internal consistency Cronbach’s alpha was computed for the 36-item scale, using 193 preceptor assessments of new RN graduate competency completed at the start and end of the transition program. Cronbach’s alpha was 0.92. For the pre-program assessment, 135 preceptor evaluations were available; 105 were complete and included in the analysis. Of 3780 potential responses to individual items, 3615 (98.7%) were completed. Ten items were completed by all respondents, and a total of 34 were missing four or fewer responses. However, item #2 (“Complete understanding and interpretation of assessment data”) and item #9 (“Establishes rapport with patient and family”) lacked 10 and 12 responses, respectively. Item #9 was repeated under a different competency as item #18, and in that position, only three responses were lacking. In all, 98.7% of potential responses were completed. For the post-program assessment, 123 preceptor evaluations were available; 88 were complete and included in the analysis. Four items were completed by all preceptors, and a total of 27 items were missing responses from four or fewer preceptors. Items #6 (“Complete understanding and interpretation of assessment data”) and #25 (“Uses appropriate language and tone when resolving conflict”) lacked five responses and item #4 (“Integrates knowledge of pathophysiology of patient conditions” lacked six responses. Two items (#3, “Able to anticipate risks related to assessment data” and #36, overall assessment of competency) lacked nine responses. Item #16 (“Locates, critically reviews and applies scientific evidence and medical literature”) lacked ten responses, and item #22 (“Documents patient assessment data in complete and timely fashion”) lacked 13 responses. In all, 96% of potential responses were completed. RN Transition Program Competency Assessment Tool 6 Face validity A request to evaluate face validity of the instrument and link to an online evaluation form was sent by one of the authors (SP) to six content experts, each of whom had demonstrated expertise in QSEN content as a designated trainer at their university or clinical facility and/or through peer-reviewed publications about QSEN-related content. In addition, each had experience working with pre-licensure students in clinical environments similar to those used in the transition program. Two content experts were unable to complete the face validity evaluation as requested due to other responsibilities, so four experts analyzed the instrument. We provided experts with the following definitions for the QSEN competency areas (Cronenwett et al., 2007; QSEN Institute, 2014):  Patient-centered care: Recognize the patient or designee as the source of control and full partner in providing compassionate and coordinated care based on respect for patient’s preferences, values, and needs.  Teamwork and collaboration: Function effectively within nursing and interprofessional teams, fostering open communication, mutual respect, and shared decision-making to achieve quality patient care.  Evidence-based practice: Integrate best current evidence with clinical expertise and patient/family preferences and values for delivery of optimal health care.  Quality improvement: Use data to monitor the outcomes of care processes and use improvement methods to design and test changes to continuously improve the quality and safety of health care systems.  Safety: Minimizes risk of harm to patients and providers through both system effectiveness and individual performance. RN Transition Program Competency Assessment Tool 7  Informatics: Use information and technology to communicate, manage knowledge, mitigate error, and support decision-making. We asked experts to answer several questions for each item: 1. Does the question clearly relate to one of the 6 QSEN competencies being evaluated? 2. In which QSEN competency content area does it fit BEST? (only one category) 3. Is the intent of the question clear? 4. Is the language of the question clear and unambiguous? 5. Is the question clear and unambiguous in its content? 6. Is the question written at an appropriate level for undergraduate nursing students? 7. Is the format of the question (eg, use of terms, specific situation cited, grammar) clear and understandable? 8. Do you suggest a change in the question format? The experts unanimously agreed about the category to which each item belonged and offered comments about three items. With respect to the first item (“Conducts comprehensive psychosocial and physical health history that includes patient’s perspective and considers cultural, spiritual, social considerations”), a reviewer noted that the descriptor “comprehensive” might lack precision but did not offer an alternative wording. With respect to the second item, the same reviewer wondered whether “complete understanding” was measurable and was concerned that preceptors’ standards for the completeness of understanding might vary. Again, no alternative wording was suggested. One reviewer was unsure whether item 16 was a reasonable expectation for a new RN graduate. She noted that, as an expert nurse, she was still trying to perfect the skill of locating, RN Tra