The Safe Environment for Every Kid (SEEK) Model: Dissemination and Implementation
Phase: NA
Conditions: Child Maltreatment
Sponsor: University of Maryland, Baltimore

 
Title: Dissemination and Implementation of the Safe Environment for Every Kid (SEEK) Model for 
Preventing Child Abuse and Neglect 
 
NCT #: NCT03642327 
 
Document Date: 6/29/2017 
 
 

Introduction to Proposal 
We greatly appreciate the reviewers’ feedback and respond here to the weaknesses noted. The review 
highlighted favorably the public health significance and innovative features as well as strengths of the 
investigative team and environment.  
1. 
Clarify importance of screening for child maltreatment (CM) in primary care, feasibility of 
implementing SEEK. SEEK targets major risk factors for CM, rather than directly screening for the 
problem. Over 7 million children are reported annually for CM, with many costly consequences for 
individuals, families and society.1 Relationships between primary care professionals (PCPs) and parents 
offer an excellent opportunity for prevention by addressing targeted psychosocial problems.2 There is 
mounting interest in addressing social determinants of health; 3,4 it is a priority of the American Academy 
of Pediatrics (AAP). 5 SEEK was developed with replication in mind. Neighborhood health center 
pediatricians gave input, while recognizing time constraints in busy practices. SEEK II was effective in 18 
private practices 6 and has been implemented in over 100 practices, confirming its feasibility. 
2. 
Changes to the SEEK model, “not really implementation of an evidence based practice.” We 
clarified that the core components of SEEK remain. This project examines effectiveness of new 
implementation (not intervention) strategies to facilitate scale-up of the evidence-based SEEK model.  
3. 
Need to measure CM to assess SEEK’s impact.  We added a specific aim (Aim 3) in this regard and a 
measure of CM-related ICD-10 codes from EHRs for all children 0-5 in all the participating practices.  
4. 
On-line training not effective in changing practitioner behavior, and limited role of CME. 
This project builds upon growing support for online training’s effectiveness when blended with 
interpersonal modalities such as webinars and collaborative learning; these are included in our MOC 
approach.7-9 It is convenient for learners and responds to the increasing need for student-centered versus 
instructor-centered learning. This study offers a good opportunity to examine the impact of this learning 
approach on PCP behavior. We agree that CME credits are a minor incentive, but it is still worth offering. 
5. 
Need to better describe Implementation strategies, theoretical or empirical support 
regarding them, and mechanisms for change. The training and implementation strategies are both 
more clearly described and rationalized. These were guided by social cognitive theory. 10 For example, role 
plays demonstrate how PCPs can address a targeted problem. Implementing the approach in one’s practice 
and participating in collaborative learning via webinars further foster self-efficacy. The transtheoretical 
model 11 helps PCPs use Motivational Interviewing to plan jointly with parents.12 By addressing 
psychosocial problems, SEEK aims to support parents, strengthen families, promote children’s health and 
safety, and thus prevent CM.2 
6. 
Need to better address risk of improved screening and detection of CM. SEEK only screens for 
risk factors rather than for CM. Our experience in 2 RCTs and with over 100,000 families has been that CM 
very rarely emerges as a concern – when discussing discipline. This is similar to what PCPs typically 
encounter in practice. The only direct contact between the researchers and parents is in the qualitative 
interview; we have developed our response to detecting possible CM then (see Human Subjects).     
7. 
Develop data analysis re. parent and EHR data, and power analyses. We now provide more 
details on analysis of the parent and EHR data, and more detailed power calculations. We also address the 
comparison of the SEEKonline vs. the Traditional paper and pencil implementation strategies and 
implementation outcomes. Limitations to our design, and the rationale for allowing selection versus 
randomization of service delivery facilitation (i.e., SEEKonline vs. Traditional) are described.  
8. 
Need questions of PCPs specific to SEEK. We clarify in our Measures section, several assessments of 
PCP perceptions specific to SEEK. The PCPQ probes thinking and practice related to the targeted problems. 
The PCP survey assesses their experience implementing SEEK. We also evaluate their training experience, 
and qualitative interviews will probe in depth their experience delivering SEEK.  
9. 
Focus on online training not innovative. We cut this as an innovation. However, despite being 
popular, there has been rather little evaluation of the effectiveness of this modality in physician training. 
10. Efficacy of SEEK uncertain, no tests of effectiveness. The 2nd RCT in18 private pediatric practices 
demonstrated SEEK’s effectiveness. Findings replicated key findings from the initial efficacy trial. The 
current study aims to evaluate SEEK’s effectiveness under two different training conditions as part of a 
Hybrid III effectiveness trial. SEEK’s effectiveness has been recognized by the Centers for Disease Control 
and Prevention, 13 the U.S. Agency for Healthcare Research and Quality,14 the AAP15 and the California 
Evidence-Based Clearinghouse as a top tier prevention model with a strong evidence base.16 
11. Confusion re. observational vs. experimental design. We have made clear that the training 
strategies will be randomized, and that practices will choose between SEEKonline and the Traditional 
implementation strategy.  Some measures involve observation of SEEK’s implementation.

Specific Aims 
Child maltreatment (CM) is a major public health problem, affecting many lives, in the short-and long-term, 
and costing individuals, families and our society dearly. In 2015, CM reports were made to child welfare on 7.2 
million children. 1 The Healthy People Goals 2020 call for reducing fatal and nonfatal CM.17 A U.S. Surgeon 
General called for making the prevention of CM a national priority. 18 Achieving this goal hinges on the broad 
implementation of evidence-based interventions, including in healthcare systems. One evidence-based practice 
(EBP) is the Safe Environment for Every Kid (SEEK) model, developed for pediatric primary care, and 
recognized by the CDC, AHRQ, the AAP and the California Evidence-Based Clearinghouse. 6,19-21 Primary 
healthcare offers an excellent opportunity to help address prevalent psychosocial problems (e.g., parental 
depression) that are risk factors for CM. By helping address targeted psychosocial problems, SEEK can 
strengthen families, support parents, and help build safe, stable and nurturing relationships. These promote 
children’s health, development and safety, help prevent CM, 6,20 and benefit the health of the US population. 
Despite its evidence-base, SEEK has not been widely adopted, leaving a large gap for the prevention of CM. 
To date only about 100 primary care clinics have adopted SEEK. To facilitate SEEK’s uptake, we aim to test the 
effectiveness of two pragmatic training strategies, while retaining the core elements of the model.  Medical 
practitioners are increasingly engaged in distance learning, such as online offerings. 7-9 Most often these are 
self-directed efforts such as viewing a presentation.  However, recently online trainings with more interactive 
modalities such as webinars and guidance during an introductory period have become popular. One example is 
the Maintenance of Certification (MOC) quality improvement (QI) project22 for SEEK, approved by the 
American Boards of Pediatrics and of Family Medicine for this purpose. The effectiveness of this collaborative 
learning approach vs. self-directed training regarding SEEK’s implementation has yet to be evaluated.  
To further facilitate ease of adoption, SEEKonline was developed to assist primary care professionals 
(PCPs) deliver the model consistently, with fidelity. This software guides PCPs through the steps of SEEK 
during a regular primary care visit, enables parents and PCPs to participate electronically, offers real time 
decision support and facilitates documentation. The effectiveness of this software in facilitating start-up and 
delivery with fidelity, relative to the “Traditional” paper and pencil approach (as in the original RCTs) has not 
yet been examined. In addition, a third of US children receive their primary care from family medicine 
practitioners, making it important to evaluate SEEK in these settings too.23 
Proposed Study. This randomized Type III hybrid design24 leverages a commitment by 5 major healthcare 
systems to implement SEEK, enabling a rigorous evaluation of implementation strategies to optimize the 
adoption and delivery of SEEK in primary care settings, and subsequent prevention of CM: the (1) independent 
online (IND) 25-27versus in-depth structured (MOC) training 22,28,29 and (2) use of SEEKonline vs. the Traditional 
approach to guide fidelity of model delivery. Further, SEEK will be examined in pediatric and family medicine 
settings, increasing the generalizability of findings. As in PAR-16-238, this proposal targets “strategies to 
implement health promotion, prevention, screening, early detection, and diagnostic interventions, as well as 
effective treatments, clinical procedures, or guidelines into existing care systems” through these aims:  
 
Aim 1: Evaluate the effectiveness of targeted implementation strategies on the implementation 
of SEEK in primary care settings. PCPs will be randomly assigned to one of two training conditions (IND 
or MOC). H1. MOC training will lead to more positive attitudes, comfort level and competence in addressing 
risk factors for CM. H2. Users of SEEKonline will deliver the intervention more often, achieving a higher rate 
of penetration, and will report higher levels of provider and parent satisfaction than the Traditional mode of 
delivery. H3. MOC training and SEEKonline will together optimize adoption and sustainment of SEEK.  
Aim 2. Evaluate the impact of inner context variables (e.g., variation between pediatric and 
family medicine) on the SEEK implementation process and understand associated barriers and 
facilitators to successful service start-up and sustainment of SEEK delivery. Using a mixed 
methods approach, standardized measures of the implementation process (Stages of Implementation 
Completion; SIC) and associated cost (Cost of Implementing New Strategies; COINS), will be integrated with 
qualitative interview data focusing on barriers and facilitators during implementation from exploration to 
sustainment. Variations in adoption, model fidelity, and sustainment, and the economic ramifications of each 
of the SEEK training and implementation strategies will be examined.    
Aim 3. Examine the effectiveness of the intervention strategies in preventing CM. CM will be 
measured via prevalence of ICD-10 codes related to CM obtained from electronic health records for all children 
0-5 attending the practices. H1. Incidence of CM will be reduced in practices after implementing SEEK. H2. 
Practices randomized to MOC training that successfully implement SEEK, will have lower incidences of CM 
than with the IND approach. We will also probe the influence of SEEKonline and the Traditional approach on 
CM rates. Additionally, implementation success, will be examined in relation to CM prevalence rates.  

A. SIGNIFICANCE      
Prevalence. Child maltreatment (CM) is a major public health problem in the US, affecting many lives, in the 
short-and long-term, and costing individuals, families and our society dearly. In 2015, 7.2 million children were 
reported to Child Protective Services (CPS) 1 Of these, 683,000 children (i.e., 9.2 per 1000) were 
“substantiated” victims of child abuse and neglect. Yet, reported cases capture only the tip of the iceberg. The 
National Incidence Study (NIS-4), using observations by community professionals, estimated that 1.26 million 
children (i.e., 17.1 per 1,000) were maltreated in 2005-06, and the more inclusive “Endangerment Standard” 
estimated nearly 3 million victims (i.e., 40 per 1,000). 30 Outcomes. The consequences of CM can be 
devastating - in the short- and long-term. In addition to injuries and physical health problems, child and 
adolescent sequelae include many psychological and behavioral problems. 31-35 CM has also been linked to an 
array of adult outcomes such as substance use disorders, HIV/AIDS-related sexual risk behaviors36-42 and being 
a victim and/or perpetrator of intimate partner violence (IPV). 43-48 Other problems in adulthood include 
depression, suicide, criminal behavior, interpersonal problems, and academic and vocational difficulties, 49-58 as 
well as multiple physical health problems.59-62 Costs. The financial costs of CM are immense. The costs 
associated with injuries due to physical abuse far exceed the per-child costs of non-inflicted injuries of 
comparable severity.63  Two-thirds of the medical costs of CM are paid through Medicaid.64 Additional costs are 
incurred by the child welfare, educational, mental health, and judicial systems, with estimated U.S. costs of 
$103.8 billion per year. 65 The human and economic costs of CM point to the serious need for effective 
preventive strategies, and the broad implementation of evidence-based practices (EBPs). 
Need for and Potential of Primary Prevention of CM. Despite the compelling need to prevent CM, few 
interventions other than home visiting programs (e.g., SafeCare) have been rigorously evaluated and found to 
be effective. Further, programs have not been developed for the healthcare system with the exception of 
preventing abusive head trauma, an important but small component of CM. The Safe Environment for Every 
Kid (SEEK) model, developed for pediatric primary care, was found to have helped prevent CM in two large 
federally f-funded, randomized controlled trials(RCTs),  and is the focus of this proposal. 6,19-21  Despite its 
success, as is often true with effective interventions, questions remain as to how best implement this model. 
The overall aims of the proposed study are to examine technology-driven approaches to implementing the 
model and to understand facilitators and barriers regarding its implementation and short-term sustainment, 
while also examining the effectiveness of these strategies. Doing so will advance knowledge in implementation 
science related to primary care and the prevention of CM. 
The prevention of CM requires strengthening families, and supporting parents and parenting, thus 
fostering safe, stable, nurturing relationships (SSNRs).66 SSNRs in turn are important for promoting children’s 
health, development, and safety, and, for preventing CM. Ordway et al pointed out the critical role that 
healthcare professionals can play in promoting healthy relationships, stating “this requires a shift in pediatric 
healthcare’s focus to include families and communities, as well as a shift to include children’s social and 
emotional health.” 66 Effective interventions in this area can achieve much more than preventing CM. 67 Olds 
and colleagues, for example, found that the Nurse Family Partnership program led to fewer perinatal 
complications, more constructive modes of discipline, increased spacing of pregnancies, fewer problems with 
the justice system, and less use of illicit drugs and tobacco, in addition to reduced CM. 68,69 Margolis et al70 
found that interventions in pediatric practices resulted in fewer childhood injuries, less maternal smoking, and 
more stimulating home environments. In the 2nd SEEK RCT, mothers in the SEEK practices reported after 12 
months fewer physical assaults by their partners than did controls (13% vs. 18%, p=.045). This helped explain 
the relationship between SEEK and fewer physical assaults of children by their mothers (1R49CE000588). 
Thus, the prevention of CM involves health promotion – for the child and family.  
Linking CM Prevention and Pediatric Primary Care. The delivery of pediatric care by primary care 
providers (PCPs; i.e., pediatric and family medicine physicians, nurse practitioners and physician assistants) is 
via regularly scheduled checkups. As outlined by the AAP, “the foundation of Bright Futures health supervision 
is health promotion – not just preventing or treating illness or injury but actively promoting the physical, 
emotional, mental, and social well-being of children, adolescents, and their families.” 71 Pediatric primary care 
is well positioned to play a valuable role: 1) it is well institutionalized, 2) prevention is a priority, 3) PCPs 
usually enjoy trusting relationships with families, 4) there are 12 visits from 0 – 5 years, 5)  there is a 
longstanding concern with children’s safety, 6) PCPs  often are the only professionals having contact with 
families of preschoolers, and 7) there is a broad commitment to children’s overall health, development and 
safety. 72,73 Haggerty et al., 73 in "The New Morbidity,” drew attention to major psychosocial problems, such as 
divorce, substance abuse and CM, jeopardizing children’s health and wellbeing. Bright Futures translated these 
ideas into practice guidance, urging consideration of a child’s environment. 71,72 Parents also are interested in 
PCPs playing this broader role. 74 Pediatric primary care’s potential was shown by  Olson et al. 75 where 40% of 
mothers who screened positive for depression accepted referrals for evaluation. Despite this, there only have 

been modest shifts in practice. Sharpe et al. 76 found that although psychosocial concerns were raised in 88% of 
checkups, pediatricians intervened in only 40%, partly due to feeling incompetent. 77 The SEEK model was 
developed to fill this critical gap and take advantage of the excellent opportunity in primary care to help PCPs 
address common psychosocial problems known to be risk-factors for CM. Given the parameters of busy 
practice settings, the need for a low-burden, yet effective intervention was clear. Working with an Advisory 
Committee of neighborhood health center physicians, we factored in the time constraints and other screening 
efforts while developing SEEK.  The two SEEK RCTs demonstrated the model’s feasibility, with over 100 PCPs 
delivering the SEEK intervention. Moreover, although limited in reach compared to the vast number of 
primary care practices in the U.S., over 100 practices have indeed successfully adopted the model 
demonstrating real-world feasibility.    
Importance of Family Medicine. While most children receive primary care in pediatric offices, about 1/3 
receive this care by family physicians. More than 2/3 of family physicians provide pediatric primary care. 23 
Family medicine provides an excellent opportunity for implementing SEEK. Psychosocial problems top every 
list of common issues addressed. Family medicine PCPs usually care for the whole family, and are thus 
particularly well positioned to assess and understand how psychosocial risk factors can affect an entire family. 
These PCPs already encounter the problems by SEEK, and need to be equipped to address them. 
Implementation Science (IS) in Pediatric Primary Care Research. The field of implementation 
science has recently experienced an “explosion” in the quality and quantity of research. 78 Yet, a review of the 
Cochrane Effective Practice and Organization of Care (EPOC; http://epoc.cochrane.org/our-reviews) website 
under Implementation Strategies revealed no systematic reviews regarding primary prevention in pediatric 
care and none on screening related to CM. The proposed research will therefore contribute to implementation 
science in critical areas. Proctor, Landsverk and colleagues introduced the distinction between “intervention 
strategies” and “implementation strategies,” and the concept of “implementation outcomes” distinct from 
“service outcomes” and “client outcomes.” 79 The current study aims are guided by this frame, 78 including 
training and mode of delivery as implementation strategies, and examining implementation outcomes (e.g.,  
adoption, feasibility, fidelity, penetration, acceptability, short-term sustainment, and costs), as well as SEEK’s 
effectiveness in preventing CM. The study will also contribute to the IS understanding of sustainment of 
innovations in service systems (e.g., pediatric primary care), an important yet little understood area in IS.80 
    Common barriers to implementing and sustaining new interventions like SEEK include costs, time 
constraints, PCP attitudes, and failure to anticipate barriers to widespread implementation. The logistics in 
gathering PCPs for in-person training are formidable. Thus, many EBPs for primary care “die on the vine,” ripe 
fruit that is never picked. Webinars and online training, however, offer convenience and appear promising. 22,25-
29 However, even when implemented, effective strategies for sustaining EBPs in primary care are often lacking 
and thus discontinued due to factors such as poor integration into the work flow and lack of reimbursement.  
   To our knowledge, no other study has reported on implementation costs associated with a CM preventive 
intervention. To assess the costs of SEEK, we will measure the related costs and practice-level variations. To 
broadly disseminate effective interventions, practice costs and barriers to implementation and sustainment 
must be addressed. This is done by training and facilitating use of the intervention for adoption and integration 
into routine practice. The use of technology to facilitate accessibility, reliability, and fidelity of model 
components is an implementation strategy that increasingly fits with the workflow in primary care practices. It 
also helps integrate services into existing administrative activities (e.g., billing). Knowing the opportunity cost 
afforded by an intervention such as SEEK can overcome barriers related to PCP reluctance to adopt a new 
intervention, even if an EBP, without knowing the impact on practice resources. Thus, quantifying the 
implementation cost of SEEK is needed.  
The Safe Environment for Every Kid (SEEK) Model (see Figure 1) 
    Scientific Basis. Two theories 
guided SEEK’s development. 
Ecological-developmental 
theory81,82 recognizes the multiple 
and interacting systems 
surrounding a child. Pediatric care 
has focused narrowly on the child; 
SEEK was based on understanding 
the strong influence of family and 
parental functioning on children’s 
health, development and safety, 
and on CM. SEEK was also guided 
by the Transtheoretical model83-85 

that links an understanding of a person’s stage of change (e.g., pre- contemplative) with an intervention 
tailored to the individual. Principles of Motivational Interviewing (MI) too have been incorporated.12,86 
Prevention science, Prevention science, integrating multiple disciplines, also guided SEEK’s development.87 
CM, with its multifactorial etiology, demands collaboration among disciplines. SEEK aims to effect such 
changes by enhancing PCPs’ abilities to address the targeted problems, working with professionals in other 
disciplines and agencies. The SEEK model also has been guided by  social cognitive theory.10 For example, role 
plays demonstrate how PCPs can address problems. Implementing these approaches, interacting with parents 
and behavioral health professionals and gaining confidence foster self-efficacy. In addition, the US Preventive 
Services Task Force recommends screening for intimate partner violence (IPV), depression, and alcohol 
misuse. 88 SEEK provides a structured approach to follow this recommendation, and help prevent CM.      
   The SEEK Model. Core components include: 1) training PCPs to identify and help address targeted   
psychosocial problems that are prevalent risk factors for CM: parental depression and major stress, substance 
abuse, IPV, harsh discipline and food insecurity; 2) the evidence-based SEEK Parent Questionnaire (PQ) to 
screen for the problems at well child visits 89-94 (see Appendix); 3) the Reflect–Empathize –Assess–Plan (REAP) 
approach to help PCPs assess and address problems; 4)  principles of Motivational Interviewing; 5) SEEK 
Parent Handouts for each targeted problem, customized with local resources;  and 6) a method for referring to 
community resources for identified problems –by the PCP or clinic team and office staff. Regarding the last 
component, SEEK practices in the RCTs had access to a social worker, although often problems were addressed 
by PCPs and office staff. A similar situation will apply in the proposed study where 60% of practices have 
behavioral health. In others, this will be done by PCPs and office staff. We think the "who" is not as important 
as the "what. This illustrates retaining core components while using strategies for scale-up in the “real world.”                 
     Facilitation of Implementation of SEEK to be Examined in the Proposed Study. The SEEK 
intervention has demonstrated effectiveness, but its widespread adoption by primary care clinics has been 
limited. To help facilitate its implementation, 
two pragmatic technology driven platforms 
have been developed to increase the 
convenience and accessibility of the model to 
busy PCPs: 1) an interactive web-based training 
that utilizes webinars and individualized 
guidance during service delivery start-up and 
(2) a SEEKonline software to facilitate delivery 
of SEEK within regular primary care checkup 
visits. This study will examine the effectiveness of these implementation strategies.  
    Training. Two training strategies will be tested (see Table 1), both increasingly used with PCPs, and 
developed in accordance with principles of adult leaning. 22,25-29  First, SEEK Maintenance of Certification 
(MOC-4) is an example of a structured Quality Improvement (QI) project approved by the American Boards of 
Pediatrics and of Family Medicine, required of physicians to maintain Board certification. It includes viewing 
the training videos and implementing SEEK in one’s practice and conducting the QI Plan-Do-Study-Act cycle. 
This PDSA cycle involves learning from data collected before and during SEEK’s implementation to assess and 
improve the process. Four 95 one-hour webinars over 4 months enable collaborative learning and mentoring.  
Second, Independent online (IND) training is a student-centered approach; it involves viewing the training 
videos over 2-3 hours and passing the post-test. Both approaches include three 1-hour consultations in the 
ensuing year and offer CME credits toward state licensure. Although both are convenient, we hypothesize that 
the interactive MOC-4 training will lead to more efficient and competent adoption of SEEK.   

        Facilitation of Service Delivery.  SEEKonline is software to efficiently implement SEEK via a secure 
web-based system interfacing with a 
practice’s EHR. SEEKonline has been 
developed and is currently being beta 
tested. Its effectiveness in improving 
implementation outcomes compared to 
the Traditional paper-and-pencil 
approach has yet to be examined. Data 
are stored in a Microsoft SQL Server 
Database, in an isolated, secure, private 
cloud environment, behind a fully 
HIPAA-certified network security device. 
SEEKonline enables parents to privately 
complete the SEEK PQ before a child’s 
checkup. Responses are available to PCPs 
at the start of the visit and there is real time decision support for PCPs (e.g., how to respond to a parent who 
thinks counseling is useless) vs. the Traditional paper-and-pencil approach.  Selected documentation is sent to 
the child’s EHR. In addition, SEEKonline provides aggregate data for needs assessments and QI projects. 
Nevertheless, some practices will choose the Traditional approach; they too will include SEEK-related 
information in a child’s EHR (see Table 2).  
 
Summary. Primary care offers an excellent opportunity to help address psychosocial problems, promote 
children’s wellbeing, and help prevent CM by broadly implementing the SEEK model. The proposed study aims 
to use technology to facilitate the accessibility and implementation of SEEK and to evaluate its effectiveness in 
helping prevent CM. This  fits well with NICHD’s vision for translational research  to promote sustainable 
behavioral change in the context of social factors, and “to transform this knowledge into effective interventions 
to improve health across different settings and populations.” 96 
 
B. INNOVATION 
This proposed study involves a rigorous hybrid III design to address both implementation and effectiveness 
aims, integrating knowledge from pediatric primary care, family medicine, CM prevention, implementation 
science and health economics, to examine how technology driven implementation strategies can enhance 
SEEK’s scalability. 
Prevention in primary healthcare settings. The proposed study will offer a new understanding of the 
barriers to and facilitators of the adoption and optimal implementation of SEEK, the only EBP for the 
prevention of CM developed for primary healthcare. 6,19-21                                                                        
Pediatric and family medicine primary care practices. SEEK will be studied in both settings. This is 
important in that studies to date have only been in pediatric practices.  
The use of technology driven implementation strategies. As medicine transitions to electronic 
approaches, this study will provide valuable information regarding online and distance learning as well as the 
SEEKonline web-based software, and how theses influences PCPs’ practice.  
Understanding PCPs’, behavioral health professionals’, office staffs’ and parents’ experiences 
implementing SEEK. The study aims to understand participants’ experiences regarding modalities 
developed to help implement the model, including the SEEK website, online training videos, and SEEKonline. 
In addition to quantitative measures, qualitative interviews with multiple groups of participants will be 
conducted to examine the effectiveness of different training and intervention strategies, in varied settings.   
Assessment of implementation process and associated costs within primary care settings. 
Although successfully implemented in our previous RCTs and 100 early adopting practices, there is much to be 
learned about the implementation process, costs and organizational factors that impact the SEEK’s 
implementation in real-world settings. The Stages of Implementation Completion (SIC) 97 provides an 
empirical way to characterize the implementation process. This framework enables a refined understanding of 
the implementation process and outcomes. The associated COINS (i.e., Cost of Implementing New Strategies) 
measures implementation costs. 98 We will look at these implementation outcomes in relation to organizational 
factors such as leadership and climate—known correlates of implementation success 99 Finally, we will employ a 
mixed-methods design that integrates qualitative data from multiple groups of participants with quantitative 
data on intervention outcomes. 100                                                                                                                     
A novel approach to tackling sensitive psychosocial problems. Aside from post-partum depression, 
SEEK is one of few screens for social determinants of health based on evidence from pediatric settings. 6,19-21               
Table 2. SEEK Intervention Strategies 
 
SEEK online 
Traditional 
Parent completes SEEK PQ online 
Parent completes SEEK PQ with paper and 
pencil 
SEEK PQ adds probes for positive screens 
Probes are conducted during the visit, orally 
PCP has parent’s info at start of visit 
PCP has parent’s info at start of visit 
Electronic decision support for PCP 
PCP has SEEK algorithms as Word documents 
Auto documentation 
PCP needs to document 
Info sent to private Care portal 
N/A 
Parent Handouts readily printed 
Parent Handouts need to be printed in advance 
Information readily integrated into EHR 
Information less readily integrated into EHR 
Aggregate data readily available for QI 
projects 
Aggregate data not readily available for QI 
projects 

Guidance for dissemination and implementation strategies. The above information should guide the 
development of new strategies to disseminate and implement evidence-based interventions such as SEEK in 
primary healthcare systems, and help build the knowledge base in IS. If successful, outcomes will be applicable 
to addressing social determinants of other child physical and behavioral health problems. 
 
C. APPROACH 
Preliminary Studies - Evidence from Two RCTs Supporting the Effectiveness of SEEK  
Two federally-funded RCTs have been conducted on SEEK over the last 15 years. 6,19-21  
Studies’ Aims. There were 2 main aims: 1) to examine whether training in SEEK would significantly 
enhance PCPs’ awareness, attitudes, level of comfort, perceived competence and practice behavior with regard 
to addressing the targeted risk factors (e.g., parental depression, substance abuse, food insecurity) for CM and 
2) to examine whether SEEK would reduce the rate of CM, compared to standard pediatric primary care.     
Studies’ Samples. The SEEK I  sample of 558 families was high risk, very low income, urban, mostly 
African American , and served by pediatric resident clinics in Baltimore, and involved 95 physicians. 19,21  SEEK 
II involved 105 pediatricians and nurse practitioners and 1,119 relatively low risk, mostly white, middle-income 
families recruited from 18 suburban private pediatric practices in central Maryland. 6,20        
Studies’ Design. Practices in both studies were randomized to either SEEK or standard care. 
Following baseline evaluations of the PCPs, those randomized to SEEK received in-person training on how to 
address targeted risk factors for CM within a primary care visit. Parents were recruited from all practices with 
initial and follow-up assessments at 6 (SEEKI) or 12 (SEEK II) months. PCPs were evaluated at 18-36 months 
after the initial training. Toward the end of the studies, after 30-43 months (SEEK I and II), the children’s 
medical records were reviewed for CM-related diagnoses, before and after implementing SEEK. Data were 
gathered from the state agency on possible Child Protective Services (CPS) involvement. We assessed parents’ 
satisfaction with the child’s PCP. In SEEK II, medical students observed PCPs during 3 checkups, at baseline 
and at study end, to observe their approach to and time spent on the targeted problems.  
Studies’ Findings  
Impact on PCPs. In both studies, PCPs in SEEK practices reported significantly greater comfort and perceived 
competence in addressing the targeted risk factors, compared to controls. 6,19-21 Improvements were sustained 
for 18-36 months. Review of medical records revealed that PCPs in SEEK practices were significantly more 
likely than controls to screen for the targeted risk factors for CM. In SEEK II, this was confirmed by direct 
observation; screening increased on average across practices from less than 5% to 62% of visits. In SEEK I, 
parents in SEEK clinics reported more favorable views of their child’s PCP. 21 Importantly, busy PCPs 
demonstrated they could effectively implement SEEK.                                                                                                                       
Impact on CM. Three measures from three sources assessed CM: 1) parent self-report on the Conflict Tactics 
Scale, 101 2) review of children’s medical records for abuse or neglect, and 3) CPS reports. Parent self-report. 
SEEK I parents reported fewer “severe physical assaults” than controls (0.11 vs. 0.33, p = .04). 19  SEEK II 
parents reported fewer instances of Psychological Aggression (p = .02) and Minor Physical Assaults (p <.05) 
than did controls. 6  Medical records. Children in SEEK I practices had less medical neglect than did controls. 19 
There was less “non-compliance” with medical care (4.6% vs. 8.4%, p = 0.05), and fewer delayed 
immunizations (3.3% vs. 9.6%, p = 0.002). CPS reports. In SEEK I, fewer families were reported to CPS (12% 
vs. 19.7%, p = .04). 19 A report was prevented in one of every 13 such families exposed to SEEK.                                             
Time Required. SEEK did not require significantly more time, on average, for PCPs to address psychosocial 
problems. 20 Parents completing the PQ before visits saved time; this was offset when addressing problems.                      
Cost. SEEK II cost $3.38 per child per year and $210 per CM experience prevented. Using a conservative 
estimate of the healthcare cost per case of CM at $2,908, providing SEEK in all practices would have saved 
society between 1.6 and 5 million dollars for 29,610 children.102 Thus, SEEK has a positive cost-benefit. 
However, it does not necessarily follow that practices can afford to implement SEEK. By focusing on 
implementation vs. intervention costs, the proposed study will evaluate the cost to a practice of implementing 
SEEK. Of note, CPT code 96161 enables PCPs to bill each time the SEEK Parent Questionnaire is administered.  
                            
Strengths and Limitations of the Extant Research on SEEK       
            Strengths. Improvements in PCP thinking and practice regarding the targeted psychosocial problems 
to be known determinants of CM were sustained in both RCTs. Indeed, there was significantly less CM 
associated with SEEK - in both RCTs. Moreover, SEEK II demonstrated effectiveness in a sample that 
otherwise, would be considered to be relatively low risk, representing many American families, suggesting the 
potential for widespread benefit from scaling up SEEK. 
Limitations Addressed in the Proposed Research. The studies did not probe what facilitated or 
challenged practices’ and PCPs’ adoption and implementation of SEEK. In addition, SEEK was only tested in 

pediatric settings; there is a need to examine it in family medicine practices. Further, the labor-intensive in-
person training used in the RCTs reduces the capacity for widespread scale-up. Evaluation of easily accessible  
methods for training and guidance in implementing SEEK including online training and SEEKonline decision 
support have the potential to increase adoption and implementation of SEEK with fidelity more broadly.  
 
Strong Multidisciplinary Team                                                                                                                                          
Dr. Dubowitz (PI) is internationally known in the field of CM and has worked closely with Drs. Landsverk, 
Magder and Herr. Dr. Herr is expert in software for healthcare settings. Drs. Landsverk, Saldana, and Palinkas 
have collaborated across multiple implementation studies, including Landsverk’s P30 Center focused on 
implementation methods (P30MH074678). They are well known dissemination and implementation 
researchers, including Saldana’s development and testing of the SIC and COINS procedures, and Dr. Palinkas’ 
expertise in mixed-methods in implementation research. Dr. Zolotar is a family medicine physician and 
experienced researcher. Dr. Bradford is a health economist with extensive experience in mental health services 
research and helped develop the COINS. Dr. Glascoe is a developmental child psychologist with extensive 
expertise in the implementation of screening tools in primary care.  
 
Strategy and Design Overview (Figure 2).   
The proposed study, based on SEEK I and II findings is designed to study the effectiveness of technology driven 
training strategies to facilitate use of SEEK in helping prevent CM and its adoption and implementation in 
pediatric and family 
medicine settings.  The 
adoption and 
implementation of EBPs 
may be even more 
challenging than their 
development and testing. 97 
This may apply especially to 
healthcare systems and 
preventive psychosocial 
interventions such as SEEK.      
Implementation 
framework: EPIS 
model. The 
implementation approach is 
anchored in four stages of 
the EPIS conceptual model: 
103 Exploration, 
adoption/Preparation, 
Implementation, and Sustainment. The strong evidence for SEEK’s effectiveness has been underscored by its 
listing on the websites of the CDC, AHRQ, AAP, and by the California Clearinghouse for Evidence-Based 
Interventions in Child Welfare. Early adopters are increasingly implementing SEEK in primary care settings, 
including in Sweden. Four of the 5 healthcare systems in this proposal have implemented SEEK in some of 
their practices. However, practices within these systems operate independently of one another, and there 
should not be contamination among them. Each practice has its own lead physician, PCPs and administrative 
staff and several have integrated behavioral health. Given that early adopters are estimated to be 10% of those 
eligible and that medical innovations can take 17 years to be adopted, 104 it is unlikely that SEEK’s potential 
public health benefit will be realized without better methods for scaling up.     
      Design. The proposed implementation study is the first involving SEEK. We selected a rigorous Hybrid 
Type III design which is an approach to examine implementation outcomes while also examining SEEK’s 
effectiveness in preventing CM, measured by EHR indicators.24 Type III designs are facilitated by good EHR 
systems due to the low cost of data routinely gathered for clinical, services, and financial purposes. This allows 
more resources to address the implementation outcomes of SEEK strategies in routine practice settings. In the 
past 4 years, over 80 papers reported use of hybrid designs. 105 Practices will be randomized to one of the two 
training strategies (IND vs. MOC). Practices, however, will be able to select the facilitation strategy 
(SEEKonline or Traditional) as we are not able to require this of participants. The design accounts for 
heterogeneity in geography, size of healthcare systems, type of primary care (pediatric and family medicine), 
and presence of integrated behavioral health (see strong support letters from healthcare systems in Texas, 
Ohio, Utah, Wisconsin and North Carolina). As shown in Figure 2, professionals, office staff and parents are 
 

nested within practices which are nested within the 5 healthcare systems. The participating systems will be 
randomized to one of two start times - 9 or 16 months from project start date, beginning with one pediatric and 
one family medicine system (cohort 1). Staggering the start time addresses the capacity limitations of the 
training and research staff. 
Sample. We will offer SEEK to the 5 healthcare systems. Inclusionary criteria are: 1) practices not already 
implementing SEEK, 2) practices providing primary care to children, and 3) agreement to participate. The 
systems have indicated a general commitment to this project; they include 59 practices with 306 PCPs. Our 
recruitment efforts in SEEK II were very successful with 75% (18/24) of practices agreeing to participate, 
including all the PCPs in those practices. We plan to study 13 pediatric and 35 family medicine practices. If 
more practices are interested, we will select the first 48 choosing to participate. In the unlikely event that fewer 
than 48 are interested, we will approach other healthcare systems. In addition to 5 system leaders, we will 
recruit 5 groups of participants: 1) all 48 practice leaders, 2) 275 PCPs, 3) 16 behavioral health professionals, 4) 
96 ancillary staff, and 5) 288 parents (6 per practice). These estimates are based on current staffing. Smaller 
subsamples will be purposively selected for qualitative interviews according to practice type, training strategy 
and presence of integrated behavioral health.  
We will also request limited consent from practice leaders and PCPs who opt out to briefly probe their 
decision. We will ascertain aggregated ICD-10CM-related diagnoses on all children 0-5 attending the practices 
(N~90,000). Based on our experience, we anticipate 60% of practice leaders and PCPs will be female as will be 
most of the other participants. Different regions provide diversity in terms of urban, suburban and rural 
locations, and racial/ethnic diversity (e.g., 2 systems serve a predominantly Latino population). Three systems 
have integrated behavioral health professionals. These contextual factors will be considered in all analyses.  
Recruitment. Leaders in the 5 healthcare systems will be approached to formally approve participation, 
without committing individual practices. A letter will be sent to physician leaders of practices inviting 
participation in the study. If interested, we will hold a 1-hour webinar with the practice leader, PCPs and 
behavioral health professionals together with key office staff. We will provide written material detailing what 
their involvement will entail. Participation will be voluntary, and, while we prefer that all the PCPs in a practice 
participate, this will not be required. We anticipate that behavioral health professionals and office staff will 
necessarily be involved in practices opting to participate. We will sign an MOU with participating practices, and 
request a limited informed consent by practice leaders and PCPs who opt out, to gather data regarding their 
decision and practice demographics. Informed consent will be obtained from all participants. This will be done 
online at the beginning of the first survey, an approach we used successfully in SEEK II. We will recruit one 
practice manager and nurse per practice. Finally, parents will be recruited via flyers describing the project in 
the waiting area. If interested, they will be asked to notify staff who will request their contact information and 
permission to convey their interest to SEEK project staff. We will contact them and explain the project by 
phone and in writing. Prior to the start of the first online survey, participant consent will be obtained.   
 
Timeline (See Table 3 for timing of data collection) 
Month 1-8:  
Hire and train project coordinator and research assistant. Randomly select first 2 healthcare 
systems (one pediatric, one family medicine). Recruit practices and randomly assign to SEEK 
training strategies; complete training. Practices select SEEKonline or Traditional strategy, 
SEEKonline connected to practices’ EHRs, computerize study protocol, obtain IRB approvals 
Month 5-48: Periodic webinars with PCPs and evaluation of participants, develop papers and presentations 
Month 9:  
Begin implementing SEEK model in cohort 1 of healthcare systems and practices 
Month 11-15:   Repeat above process with cohort 2 of healthcare systems and practices 
Month 16: 
Begin implementing SEEK model in cohort 2 
Month 46-48: End of observation period, practices decide whether to continue using SEEK, final surveys 
Month 49-60: Data will be obtained from practices’ EHRs re. implementation (e.g., rate of screening, 
CM-related ICD-10 codes), final data analyses, prepare Final Report and papers  
Procedures  
Pre-implementation Phase (5 months). Recruitment will begin and we will identify a physician “champion” 
and an office staff member to lead implementation in each practice. Practices will choose by month 4 
SEEKonline or the Traditional paper and pencil approach. To ease introduction of the SEEK model, we will 
address logistical issues such as for which checkups parents will complete the SEEK PQ and documentation in 
the EHR. SEEK Parent Handouts will be customized with information on local resources and PCPs will 
complete the SEEK Primary Care Provider Questionnaire (PCPQ) prior to the SEEK training.                                                
Implementation Phase. In month 5, practices in cohort 1 will be randomized to one of the two SEEK 
training strategies (MOC or IND), stratified by practice type (pediatric or family medicine). At the end of 
training in month 8, PCPs will evaluate it. After the training, PCPs will be asked to participate in 3 webinars 

every 4 months to discuss implementation and clinical issues. By month 8, interested practices will have 
SEEKonline connected to their EHR, ensuring interoperability. In month 9, the SEEK intervention will begin in 
cohort 1. Practice leaders, PCPs, behavioral health professionals and office staff will be asked to complete the 
online SEEK Adoption Survey, regardless of full participation. Parents also will begin completing surveys. 
Purposive subsamples from each group will be selected for 30-minute phone qualitative interviews. The 
process for cohort 2 will begin in month 11 and the SEEK intervention will start in month 16. Procedures are 
detailed under Human Subjects for addressing concerns of possible child abuse or neglect. 
Sustainment Phase. Participants will be assessed at 12, 21 and 33 months post baseline regarding their 
thinking, feeling, behavior and experiences (see Table 3) to help inform understanding of SEEK’s 
implementation and sustainment. At months 41 and 48, the observation period will end for cohorts 1 and 2. 
Final surveys and phone interviews will assess participants’ thoughts and plans about continued use of SEEK. 
Following the grant period, practices that wish to continue utilizing the implementation strategies will be 
guided through the process of establishing real-world, not-grant funded contracting. 
Final 12 months. De-identified aggregate data will be obtained from practices’ EHRs on all children (0-5) for 
the periods prior to and during SEEK implementation: rate of eligible visits where screening occurred, types of 
problems identified, types of actions taken, receipt of services, as well as  CM-related diagnoses using ICD-10 
codes. This will be facilitated by 4 of the 5 systems using EPIC as their EHR and having in-house IT. The data 
will be analyzed, the Final Report prepared, and presentations and papers will continue to be developed. 
 
Measures (See Table 3) 
Measurement will include survey-based assessments, targeted qualitative interviews, observational 
implementation assessment, and EHR data on service delivery of SEEK (e.g., rate of screening) and CM.  
Organizational Assessment 
Organizational measures will be administered at baseline regarding the opportunity to adopt SEEK. Practice 
leaders and PCPs who decide to participate in the study will be recruited; those who opt out will be asked to 
complete the one-time brief online SEEK Adoption Survey and the following 4 measures. A subsample will 
undergo a semi-structured, 30-minute phone interview.  
Demographics Form. This includes the healthcare system and practice structure, annual budget, number of 
practices, number and characteristics of PCPs, presence of behavioral health and patient demographics. 
Evidence-Based Practice Attitude Scales (EBPAS). 106 The 15-item EBPAS measures 4 domains of 
service providers’ attitudes toward adopting EBPs: appeal (the extent to which one would adopt an EBP if it is 
appealing), requirements (willingness to adopt an EBP if required to do so), openness (general openness to 
new interventions), and divergence (the degree to which one believes that EBPs are less important than clinical 
experience). Subscales have adequate internal consistency (Cronbach’s α = .67-.91) 107 and convergent validity.  
Implementation Leadership Scale (ILS). 99 The 12-item ILS assesses leader support for EBPs within an 
organization (5-point Likert scale). Four subscales - proactive, knowledgeable, supportive, and perseverant 
leadership - have excellent internal consistency (α = .95 to .96), and convergent and discriminant validity. 
Implementation Climate Scale (ICS). 108 The 18-item ICS assesses how the organization views new 
interventions. Implementation climate is defined as shared perceptions of policies, practices, procedures and 
behaviors that are rewarded, expected and supported. The measure has strong psychometrics with internal 
consistency (α = .81-.91), convergent validity, and validation in a sample of child welfare professionals. 108      
Implementation Process and Outcomes 
The Stages of Implementation Completion (SIC) is an observational assessment tool developed in a 
large randomized implementation trial. 38,97,109 The SIC has 8 stages, each including sub-activities, extending 
from Engagement with the developers to achievement of practitioner Competency. Each stage maps onto 3 
phases of implementation (Pre-Implementation, Implementation and Sustainment). Competency is viewed as 
entering into the Sustainment phase. Sustainment is an ongoing process toward achievement of Stage 8.   
 
The SIC has been adapted for over 20 different implementation strategies involving varied interventions. 
These multiple adaptations led to the development of the empirically derived Universal SIC, with items found 
to be relevant and reliably utilized across each of the adaptations, regardless of service sector or population 
(e.g., child welfare, primary care, prevention). The Universal SIC spans the 8 original SIC stages, with 45 
activities tracked during implementation. For the current study, completion of activities will be monitored by 
the practice manager and collected monthly by the research team, with data entered into the SIC data 
collection website. The website is managed by the SIC team, and through a web-based tool, implementation 
performance at the practice level can be benched against other implementation efforts.  
SIC Scores. Three scores are calculated for each SIC stage. First, the time that a practice takes for a stage is 
calculated (Duration Score). Because the implementation process is nonlinear, the Duration Score takes into 
account activities that may not be completed sequentially within a stage. If a site completes only one activity in 

a stage, the Duration Score for that stage is a single day. Duration Scores are calculated separately for each 
stage due to the potential for overlap across stages; the total Duration Score does not necessarily equal the 
summation of each stage’s Duration. Rather, the total Duration Score is calculated from the date of the first 
completed SIC activity to the final activity completed, regardless of the stage in which it occurred. Second, the 
percentage of activities completed within a stage is calculated Proportion Score). Similar to Duration, 
Proportion Scores are calculated 
separately for each stage. The total 
Proportion Score is based on the 
number of activities completed out 
of the number that could be 
completed – for each stage. 
Finally, the SIC Stage Score marks 
the final stage that a site reaches. 
SIC scores are calculated within 
each of the three Implementation 
Phases: Pre-implementation 
(stages 1-3), Implementation 
(stages 4-7), and Sustainment 
(stage 8).  
SEEK Targeted Adoption, 
Training, Implementation 
Assessment                                                   
SEEK Adoption Survey. This 
survey will capture key 
considerations regarding decisions 
whether to adopt SEEK, including 
perceived barriers and strengths.                   
 
SEEK PCP Training Evaluation Form. This online survey will be developed to evaluate PCPs’ perceptions 
of the training approaches and solicit input as to how the training might be improved.                                                             
SEEK PCP Questionnaire. (PCPQ). The PCPQ, used in both SEEK RCTs 20,21, has 5 vignettes, with 7-12 
statements assessing PCPs’ thinking and practice with regard to addressing the targeted CM risk factors. Each 
vignette is followed by statements such as “I know how to motivate parents who may be resistant to 
suggestions” rated on a Likert scale. Items were grouped conceptually into 4 topical scales (e.g., Substance 
Abuse) and also cross-cutting themes (e.g., Perceived Competence). Cronbach’s 107 alphas were adequate for 
most scales: Depression (.76), IPV (.80), Major Stress (.80) and Substance Abuse (.58), and for most themes: 
Attitudes (.77), Knowledge (.55), Comfort (.68), Competence (.74) and Practice (.70).                                                               
SEEK PCP Survey. This 19-item measure uses a 5-point Likert scale to assess PCPs’ experience 
implementing SEEK including their  perceptions of its relevance, ease of delivery, helpfulness and 
training.                                                                                                                                             SEEK Office Staff 
Survey. This 23-item measure uses a 5-point Likert scale to assess staff experiences with SEEK, including how 
they understood their roles and perceived competence in implementing SEEK.                                                                       
Rate of Screening. EHR data on the proportion of eligible visits where screening occurred, in a given period. 
Receipt of Services. EHR data on referrals and receipt of services by parents with positive screens.                                 
SEEK Parent View of Child’s PCP. Parents rate their child’s PCP on this measure adapted for pediatric 
practice from the Patient-Doctor Interaction Scale; reliability and validity are good. 110 The Likert scale was 
changed to a yes-no response. Individual scores range from 0 – 20; higher scores indicate greater satisfaction.   
Child Maltreatment 
EHR CM-related Diagnoses. ICD-10 codes related to CM are readily accessible through EHRs. 111 De-
identified aggregate data will be gathered toward study end for all children 0-5 attending the practices during 
the study – for up to 2 years prior to and during the study.  
Qualitative Assessment: Phone Interviews                                                                                                              
To assess feasibility and acceptability and potential barriers and facilitators of the SEEK intervention, we will 
conduct 30-minute semi-structured phone interviews with purposively selected (based on role, practice type, 
training strategy and presence of behavioral health) subsamples of 5 groups: practice leaders, PCPs, behavioral 
health professionals, office staff and parents, at 3 time points. For the first 3 groups, these will be around the 
start of the intervention (probing adoption) and 11 and 21 months later. For the last 2 groups, interviews will be 
at 11, 20 and 32 months following start of the intervention, after they have experienced SEEK. Interviewers will 

use a semi-structured guide similar to one previously employed. 112 Participants will be asked about their 
experiences with SEEK, assessment of training and implementation support, challenges in delivering the 
intervention and recommendations for addressing the challenges. Interviews will be conducted by telephone 
and will be digitally recorded and professionally transcribed for analysis.                                                                                     
 
 
Measuring Costs                                                                                                                                          
The cost analysis will provide estimates of the overall practice-level costs associated with implementing SEEK. 
Cost measurement will be organized using the Cost of Implementing New Strategies (COINS) framework which 
provides a structure for measuring and categorizing costs. 98 Implementation costs include all resources used to 
deliver SEEK. COINS maps onto the SIC, by tracking the costs and resources needed to complete each 
implementation activity. In SEEK II, PCPs in the intervention arm did not require more time per child than did 
controls. It seems reasonable to apply this to the current study, rather than conduct another time study. 
 
Data Analysis Plan 
General Considerations. Standard statistical methods for calculating point estimates, confidence intervals, 
and p-values require the assumption of independence. However, due to participants being clustered within 
healthcare systems and within practices, this assumption cannot be made. To account for the lack of 
independence among multiple measures within the same system or practice, we will use random effects (i.e., 
hierarchical) models. In these models, we will include a random effect for system and for practice. In analyses 
involving multiple measures from the same person, we will include a random effect for each one. These models 
can be fitted using SAS Proc GLIMMIX, which fits models for both quantitative and binary outcomes. There 
will be diverse analyses based on the data. Below we highlight the main analyses for each Aim. The qualitative 
data (see below) will inform the models through processes of convergence (i.e., triangulation) and expansion 
(i.e., explanation of unexpected findings from analyses of quantitative data).100 
    Sex as a biological variable will be considered across multiple levels of analyses. At the PCP, parent, and child 
levels of analyses, there is the potential for variation in implementation outcomes related to sex. Analyses will 
include a test for sex differences; when differences are noted, additional analyses will examine the influence of 
additional contextual factors on these outcomes.  
Aim 1. Compare alternative approaches to implementation of SEEK with respect to clinical and 
implementation outcomes.  To address aim 1a, we will compare practices randomized to MOC to practices 
randomized to IND with respect to clinical and implementation outcomes.  These include perceptions of the 
training from the PCPs’ evaluations, scales from the SEEK PCPQ (e.g., Competence in addressing problems) 
and the PCP Survey (e.g., Ease of Delivery. We will also compare the groups with respect to rates of screening 
and parents’ receipt of services and satisfaction with the PCP. For quantitative outcomes, statistical inference 
will be based on linear regression models; for binary outcomes, inference will be based on logistic regression 
models, accounting for repeated measures within PCP and system. For example, for PCPs reported 
Competence we will fit this model: E(Competenceijkl) = β0 + si + pj + ik + β1(POST) + β2(POST)(MOC-4) where 
Competenceijkl stands for the competence score measured at the lth time point (l=1 or 2 for baseline or follow-
up respectively) of the kth PCP in the jth practice of the ith system, and  si , pj , ik are random effects for system, 
practice and PCP respectively; POST=1 if it is a post-training measure and 0 otherwise, MOC = 1 if the PCP was 
trained using MOC and 0 if IND, and β’s are parameters to be estimated. Note this model assumes that the 
mean baseline scores are the same in both groups, as recommended for randomized studies. 113 This model can 
be fitted using restricted maximum likelihood. To avoid possible biases due to selective attrition from the 
training, the primary analysis will follow the “intention to treat” principle including all those randomized.  
Secondary analyses will be based on groups defined by training received.  To address Aim 1b we will compare 
practices who choose SEEK-online to practices who choose the traditional approach using the same statistical 
methods as described for Aim 1a. Outcomes of interest will include staff satisfaction, costs, rates of screening, 
receipt of services, and parental satisfaction.                                                                                                                                         
Aim 2. Examine variations in SEEK’s implementation process and impacts and understand 
associated barriers and facilitators in pediatric and family medicine practices. We will determine 
the proportion of practices that agree to adopt SEEK. We will then quantitatively assess the relationship 
between practice characteristics (e.g., demographics, EBP attitudes) and willingness to adopt SEEK. The most 
important independent predictors will be determined using multivariable logistic regression models. Among 
those that do adopt SEEK, we will assess their degree of completeness, speed, and quality of implementation 
using the SIC.  Completeness will be summarized using the final stage attained (0-8).  Speed will be 
summarized by the duration in each stage, and quality by the proportion of activities performed at each stage, 
and overall. The distribution of these variables will be determined, overall and separately, based on the type of 

practice. The distribution of time to achieve each phase will be estimated using the Kaplan-Meier approach. 
Barriers and facilitators to implementing SEEK will be probed quantitatively and qualitatively (see below). 
Among practices adopting SEEK, we will examine the association between practice characteristics and 
measures of completeness, speed, and quality of implementation from the SIC. We will examine relationships 
between practice characteristics and implementation by comparing the distribution of SIC scores in groups 
defined by practice characteristics. Finally, the most important predictors of degree of implementation will be 
determined using multivariable regression models with practice characteristics as predictors and SIC measures 
as outcomes. In addition to studying implementation at the practice level, we will analyze the relationship 
between PCP level measures of implementation (e.g., screening rates) and PCP characteristics, facilitators and 
barriers.  These models will account for clustering of PCPs within practices, as described above.   
Aim 3. Examine the effectiveness of SEEK in reducing CM. As in our previous studies, 6,19 we will 
identify CM-related diagnoses, now via EHRs.  We will review the EHR for each child (0-5) in each practice for 
two years prior to implementing SEEK and during implementation.  The presence of CM-related ICD-10 codes 
will be recorded. The proportion of children with CM diagnoses before and during SEEK will be compared at 
each practice. Formal inference regarding the best estimate and statistical significance of pre-post differences 
will be based on a binary regression model with a random effect for site, similarly to Aim 1.  
Sample Size Considerations. Our projected sample sizes (numbers of practices, professionals within each 
practice, and children in the practices) should provide good precision to estimate parameters and ample power 
to detect moderate associations. Table 4 below shows the detectable effect sizes for various comparisons, based 
on performing 2-sided .05-level tests and incorporate a 1.5-fold increase in variance and sample size to account 
for potential loss of power due to clustering of observations within practices and systems.  In general, the 
detectable effect sizes are moderate, at plausible levels. We observed effect sizes of about 1.0 regarding 
Competence and Comfort in SEEK II. For Aim 2, assuming 90% of practices provide organizational 
information, we will estimate the rate of adoption of SEEK accurately, + 15 percentage points. 
 
Cost Analysis. We will conduct cost 
effectiveness analyses taking the 
perspective of a practice. Total practice 
cost of implementation will be 
calculated for both training arms (IND 
and MOC). Cost effectiveness ratios will 
be calculated as the cost per average SIC 
component completed, and per average 
Competence scale and Practice Behavior 
scale scores. Standard time discounting 
methods will be applied to cost 
estimates. 114 We will use sensitivity 
analyses to derive upper and lower 
estimates of resource use and 
intervention costs. 114 Standard errors for 
use in mean comparisons will be 
estimated, using bootstrapping 
methods. 114 
Qualitative/Mixed Methods Analysis. An audit trail of data collected and memos, team meetings 
indicating time, place, source of data, and persons collecting or analyzing information, will be kept. Procedures 
for data collection and the role of the investigator collecting the data will be detailed and reviewed with each 
member of the research team. We will analyze interview transcripts using a thematic content analysis 
methodology. 115 First, transcripts will be distributed among investigators. Each will review these to develop a 
broad understanding of content related to the project’s aims and to identify topics for discussion and 
observation. During this and  subsequent steps, investigators will prepare short descriptive statements or 
“memos” to document initial impressions of topics and themes and their relationships, and to define the 
boundaries of specific codes (i.e., the inclusion and exclusion criteria for assigning a specific code). 116 Second, 
transcripts will be independently coded to condense the data into analyzable units. Segments of text ranging 
from a phrase to several paragraphs will be assigned codes based on a priori (i.e., from the interview guide) or 
emergent themes (or, open coding 117). Codes will be assigned to describe connections between categories and 
between categories and subcategories (i.e., axial coding 117). Codes will also be assigned to reflect the social and 
demographic characteristics of study participants. Lists of codes developed by each investigator will be 
matched and integrated into a single codebook. Third, each text will be independently coded by at least two 
investigators. Disagreements in assignment or description of codes will be resolved through discussion 

between investigators and by refining definitions of codes. The final list of codes or codebook, constructed 
through team consensus, will consist of a numbered list of themes, issues, accounts of behaviors, and opinions 
that relate to evidence use and its determinants. With the final coding structure, two investigators will 
separately review transcripts to determine level of agreement in the codes. A level of agreement ranging from 
66-97% depending on level of coding (general, intermediate, specific), indicates good reliability in qualitative 
research. 118 Fourth, based on these codes, the computer program QSR NVivo 119 will generate a series of 
categories arranged in a treelike structure connecting text segments grouped into separate categories of codes 
or “nodes.” These nodes and trees will be used to further the process of axial or pattern coding to examine the 
association between different a priori and emergent categories. They will also be used in selective coding of 
material to identify the existence of new, previously unrecognized categories. The number of times these 
categories occur together, either as duplicate codes assigned to the same text or as codes assigned to adjacent 
texts in the same conversation, will be recorded, and specific examples of co-occurrence illustrated with 
transcript texts. Fifth, by constantly comparing these categories with each other, the different categories will be 
further condensed into broad themes using a format that places SEEK’s effectiveness and implementation 
within the framework of the system characteristics. 120  Finally, the theme will be compared with the results of 
the analysis of quantitative data relating to PCP experiences specific to SEEK and its implementation to 
identify points of convergence and divergence (triangulation) and to explain or account for potentially 
unanticipated findings (expansion). 
Sample Size Considerations. Since this qualitative analysis involves no hypothesis testing using statistical 
procedures, the number of subjects necessary to produce sufficiently valid and reliable results cannot be 
calculated using standard formulas for statistical analysis. Nevertheless, previous studies relying upon this 
methodology have typically found that information becomes repetitive with little new information gained after 
analyses of data from 20-30 participants 116,117. Some authors suggest that saturation can be achieved with as 
few as 12 respondents. 121,122 By this standard, we should have an adequate sample to assess feasibility and 
acceptability and to identify potential barriers to and facilitators of implementing SEEK.  
 
Study Limitations. First, we considered randomizing SEEKonline vs. the Traditional approach, but think 
this may be difficult for some practices. Nevertheless, we will be able to compare outcomes related to these 
approaches, and, we are positioned to assess this intervention in real world circumstances. Second, while SEEK 
originally included a social worker, the core element was a means of facilitating referrals to community 
resources – often done by a PCP and office staff, and that remains. We think the “what” gets done is more 
important than the “who.” We will be able to qualitatively assess the role of behavioral health; the numbers 
however preclude doing so quantitatively. Third, medical records clearly offer only a partial view of possible 
CM, although they are a means of detecting abuse and neglect that may not be reported to CPS. Given the 
multiple regions and logistics of this study, gathering CPS and parental self-report data on CM are not feasible.  
 
Summary. This innovative hybrid III implementation trial examines two technology driven implementation 
strategies to scale-up SEEK in primary care practices. Using technology to facilitate quality training and service 
delivery of the SEEK model has the potential to increase efficiency and acceptability. Understanding how to 
best implement SEEK should enhance primary care in both pediatric and family medicine settings by better 
addressing prevalent psychosocial problems. Doing so can improve the wellbeing of families and parents and 
thereby promote children’s health, development and safety, and help prevent CM. Outcomes can provide 
valuable lessons for other implementation efforts that address social determinants of child health problems.  
 
 

 
References 
1. 
U.S. Department of Health & Human Services, Administration on Children, Youth and Families, 
Children’s Bureau. Child Maltreatment 2015. 2017. 
2. 
Dubowitz H. The Safe Environment for Every Kid Model: Promotion of Children’s Health, 
Development, and Safety, and Prevention of Child Neglect. Pediatric Annals. 2014;43(11):e271-e277. 
3. 
Alberti PM, Sutton KM, Cooper LA, Lane WG, Stephens S, Gourdine MA. Communities, Social Justice, 
and Academic Health Centers. Academic Medicine. 2017;20:00-00. 
4. 
Pettignano R, Bliss L, McLaren S, Caley S. Interprofessional Medical-Legal Education of Medical 
Students: Assessing the Benefits for Addressing Social Determinants of Health. Academic Medicine. 
2017. 
5. 
American Academy of Pediatrics. Screening Technical Assistance & Resource (STAR) Center. 2017. 
Available at: aap.org/screening. Accessed on 6/15/17. 
6. 
Dubowitz H, Lane WG, Semiatin JN, Magder LS. The SEEK model of pediatric primary care: can child 
maltreatment be prevented in a low-risk population? Academic Pediatrics. 2012;12(4):259-268. 
7. 
Anglin GJ, Morrison GR. An analysis of distance education research: Implications for the instructional 
technologist. Quarterly Review of Distance Education. 2007;1(3):189-194. 
8. 
Thompson MM, Irele ME. Evaluating distance education programs. In: MG Moore WA, ed. Handbook 
of distance education. Mahwah, NJ: Lawrence Erlbaum; 2003:567-584. 
9. 
Yang F, Wang M, Shen R, Han P. Community-organizing agent: An artificial intelligent system for 
building learning communities among large numbers of learners. Computers & Education. 
2007;49(2):131-147. 
10. 
Bandura A. Social cognitive theory: An agentic perspective. Annual review of psychology. 2001;52(1):1-
26. 
11. 
Prochaska JO, DiClemente CC, Norcross JC. In search of how people change: Applications to addictive 
behaviors. American Psychologist. 1992;47(9):1102. 
12. 
Miller WR, Rollnick S. Motivational interviewing: Helping people change. 3rd ed: Guilford press; 
2013. 
13. 
Centers for Disease Control and Prevention. http://www.cdc.gov/violenceprevention/pdf/can-
prevention-technical-package.pdf. Accessed 6/15/17. 
14. 
U.S. Agency for Healthcare Research and Quality. 
http://www.innovations.ahrq.gov/content.aspx?id=3205.  Accessed 6/15/17. 
15. 
American Academy of Pediatrics. http://brightfutures.aap.org/tool_and_resource_kit.html. 
Accessed 6/15/17. 
16. 
California Evidence-Based Clearinghouse. http://www.cebc4cw.org/program/safe-environment-for-
every-kid-seek-model/detailed. Accessed 6/15/17. 
17. 
Office of Disease Prevention and Health Promotion. www.healthypeople.gov. Accessed 6/19/17. 
18. 
Surgeon General's Workshop on Making Prevention of Child Maltreatment a National Priority: 
Implementing Innovations of a Public Health Approach: Lister Hill Auditorium, National Institutes of 
Health, Bethesda, Maryland, March 30-31, 2005 . Office of the Surgeon General (US). Rockville (MD). 
19. 
Dubowitz H, Feigelman S, Lane W, Kim J. Pediatric primary care to help prevent child maltreatment: 
the Safe Environment for Every Kid (SEEK) Model. Pediatrics. 2009;123(3):858-864. 
20. 
Dubowitz H, Lane WG, Semiatin JN, Magder LS, Venepally M, Jans M. The safe environment for every 
kid model: impact on pediatric primary care professionals. Pediatrics. 2011;127(4):e962-e970. 
21. 
Feigelman S, Dubowitz H, Lane W, Grube L, Kim J. Training pediatric residents in a primary care clinic 
to help address psychosocial problems and prevent child maltreatment. Academic Pediatrics. 
2011;11(6):474-480. 
22. 
Riley M, Ahmed S, Lane JC, Reed BD, Locke A. Using Maintenance of Certification as a Tool to Improve 
the Delivery of Confidential Care for Adolescent Patients. Journal of Pediatric and Adolescent 
Gynecology. 2017;30(1):76-81. 
23. 
Makaroff LA, Xierali IM, Petterson SM, Shipman SA, Puffer JC, Bazemore AW. Factors influencing 
family physicians’ contribution to the child health care workforce. The Annals of Family Medicine. 
2014;12(5):427-431. 
24. 
Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: 
combining elements of clinical effectiveness and implementation research to enhance public health 
impact. Medical Care. 2012;50(3):217. 

25. 
Dolan BM, Yialamas MA, McMahon GT. A randomized educational intervention trial to determine the 
effect of online education on the quality of resident-delivered care. Journal of Graduate Medical 
Education. 2015;7(3):376-381. 
26. 
Donovan AK, Wood GJ, Rubio DM, Day HD, Spagnoletti CL. Faculty Communication Knowledge, 
Attitudes, and Skills Around Chronic Non-Malignant Pain Improve with Online Training. Pain 
Medicine. 2016;17(11):1985-1992. 
27. 
Lannon CM, Peterson LE. Pediatric collaborative networks for quality improvement and research. 
Academic Pediatrics. 2013;13(6):S69-S74. 
28. 
Peterson LE, Blackburn B, Puffer JC, Phillips RL. Family physicians’ quality interventions and 
performance improvement for hypertension through maintenance of certification. Journal for 
Healthcare Quality. 2014. 
29. 
Wiggins RE, Etz R. Assessment of the American Board of Ophthalmology’s Maintenance of Certification 
Part 4 (Improvement in Medical Practice). JAMA Ophthalmology. 2016;134(9):967-974. 
30. 
Sedlak AJ, Mettenburg J, Basena M, Peta I, McPherson K, Greene A. Fourth national incidence study of 
child abuse and neglect (NIS-4). Washington, DC: US Department of Health and Human Services. 
Retrieved on July. 2010;9:2010. 
31. 
Bennett EM, Kemper KJ. Is abuse during childhood a risk factor for developing substance abuse 
problems as an adult? Journal of Developmental & Behavioral Pediatrics. 1994;15(6):426-429. 
32. 
Brown GR, Anderson B. Psychiatric morbidity in adult inpatients with childhood histories of sexual and 
physical abuse. The American journal of psychiatry. 1991;148(1):55. 
33. 
Duncan RD, Saunders BE, Kilpatrick DG, Hanson RF, Resnick HS. Childhood physical assault as a risk 
factor for PTSD, depression, and substance abuse: Findings from a national survey. American Journal 
of Orthopsychiatry. 1996;66(3):437-448. 
34. 
Glover NM, Janikowski TP, Benshoff JJ. The incidence of incest histories among clients receiving 
substance abuse treatment. Journal of Counseling & Development. 1995;73(4):475-480. 
35. 
Ireland T, Widom CS. Childhood victimization and risk for alcohol and drug arrests. International 
Journal of the Addictions. 1994;29(2):235-274. 
36. 
Johnsen LW, Harlow LL. Childhood sexual abuse linked with adult substance use, victimization, and 
AIDS-risk. AIDS Educ Prev. Feb 1996;8(1):44-57. 
37. 
Kang S-Y, Magura S, Laudet A, Whitney S. Adverse effect of child abuse victimization among substance-
using women in treatment. Journal of Interpersonal Violence. 1999;14(6):657-670. 
38. 
Morrow KB, Sorell GT. Factors affecting self-esteem, depression, and negative behaviors in sexually 
abused female adolescents. Journal of Marriage and the Family. 1989:677-686. 
39. 
Rotheram‐Borus MJ, Mahler KA, Koopman C, Langabeer K. Sexual abuse history and associated 
multiple risk behavior in adolescent runaways. American Journal of Orthopsychiatry. 1996;66(3):390-
400. 
40. 
Runtz M, Briere J. Adolescent “acting-out” and childhood history of sexual abuse. Journal of 
Interpersonal Violence. 1986;1(3):326-334. 
41. 
Sansonnet-Hayden H, Haley G, Marriage K, Fine S. Sexual abuse and psychopathology in hospitalized 
adolescents. Journal of the American Academy of Child & Adolescent Psychiatry. 1987;26(5):753-757. 
42. 
Sheridan MJ. A proposed intergenerational model of substance abuse, family functioning, and 
abuse/neglect. Child Abuse & Neglect. 1995;19(5):519-530. 
43. 
Alexander PC. Childhood trauma, attachment, and abuse by multiple partners. Psychological Trauma: 
Theory, Research, Practice, and Policy. 2009;1(1):78. 
44. 
Banyard VL, Williams LM, Saunders BE, Fitzgerald MM. The complexity of trauma types in the lives of 
women in families referred for family violence: Multiple mediators of mental health. American Journal 
of Orthopsychiatry. 2008;78(4):394-404. 
45. 
Ehrensaft MK, Cohen P, Brown J, Smailes E, Chen H, Johnson JG. Intergenerational transmission of 
partner violence: a 20-year prospective study. Journal of consulting and clinical psychology. 
2003;71(4):741. 
46. 
Halpern CT, Spriggs AL, Martin SL, Kupper LL. Patterns of intimate partner violence victimization 
from adolescence to young adulthood in a nationally representative sample. Journal of Adolescent 
Health. 2009;45(5):508-516. 
47. 
Herrenkohl TI, Mason WA, Kosterman R, Lengua LJ, Hawkins JD, Abbott RD. Pathways from physical 
childhood abuse to partner violence in young adulthood. Violence and Victims. 2004;19(2):123-136. 
48. 
Lang AJ, Stein MB, Kennedy CM, Foy DW. Adult psychopathology and intimate partner violence among 
survivors of childhood maltreatment. Journal of Interpersonal Violence. 2004;19(10):1102-1118. 

49. 
Chapman DP, Whitfield CL, Felitti VJ, Dube SR, Edwards VJ, Anda RF. Adverse childhood experiences 
and the risk of depressive disorders in adulthood. Journal of Affective Disorders. 2004;82(2):217-225. 
50. 
Malinosky-Rummell R, Hansen DJ. Long-term consequences of childhood physical abuse. 
Psychological Bulletin. 1993;114(1):68. 
51. 
Perez CM, Widom CS. Childhood victimization and long-term intellectual and academic outcomes. 
Child Abuse & Neglect. 1994;18(8):617-633. 
52. 
Dube SR, Anda RF, Felitti VJ, Chapman DP, Williamson DF, Giles WH. Childhood abuse, household 
dysfunction, and the risk of attempted suicide throughout the life span: findings from the Adverse 
Childhood Experiences Study. JAMA. 2001;286(24):3089-3096. 
53. 
Linder JR, Collins WA. Parent and peer predictors of physical aggression and conflict management in 
romantic relationships in early adulthood. Journal of Family Psychology. 2005;19(2):252. 
54. 
Smith PH, White JW, Holland LJ. A longitudinal perspective on dating violence among adolescent and 
college-age women. American Journal of Public Health. 2003;93(7):1104-1109. 
55. 
Spriggs AL, Halpern CT, Martin SL. Continuity of adolescent and early adult partner violence 
victimisation: association with witnessing violent crime in adolescence. Journal of Epidemiology and 
Community Health. 2009;63(9):741-748. 
56. 
Taft CT, Schumm JA, Marshall AD, Panuzio J, Holtzworth-Munroe A. Family-of-origin maltreatment, 
posttraumatic stress disorder symptoms, social information processing deficits, and relationship abuse 
perpetration. Journal of Abnormal Psychology. 2008;117(3):637. 
57. 
White HR, Widom CS. Intimate partner violence among abused and neglected children in young 
adulthood: The mediating effects of early aggression, antisocial personality, hostility and alcohol 
problems. Aggressive Behavior. 2003;29(4):332-345. 
58. 
Whiting JB, Simmons LA, Havens JR, Smith DB, Oka M. Intergenerational transmission of violence: 
The influence of self-appraisals, mental disorders and substance abuse. Journal of Family Violence. 
2009;24(8):639-648. 
59. 
Arnow BA, Hart S, Scott C, Dea R, O’connell L, Taylor CB. Childhood sexual abuse, psychological 
distress, and medical use among women. Psychosomatic Medicine. 1999;61(6):762-770. 
60. 
Dube SR, Felitti VJ, Dong M, Giles WH, Anda RF. The impact of adverse childhood experiences on 
health problems: evidence from four birth cohorts dating back to 1900. Preventive Medicine. 
2003;37(3):268-277. 
61. 
Farley M, Patsalides BM. Physical symptoms, posttraumatic stress disorder, and healthcare utilization 
of women with and without childhood physical and sexual abuse. Psychological Reports. 
2001;89(3):595-606. 
62. 
Felitti VJ, Anda RF, Nordenberg D, et al. Relationship of childhood abuse and household dysfunction to 
many of the leading causes of death in adults: The Adverse Childhood Experiences (ACE) Study. 
American Journal of Preventive Medicine. 1998;14(4):245-258. 
63. 
Libby AM, Sills MR, Thurston NK, Orton HD. Costs of childhood physical abuse: comparing inflicted 
and unintentional traumatic brain injuries. Pediatrics. 2003;112(1):58-65. 
64. 
Florence C, Brown DS, Fang X, Thompson HF. Health care costs associated with child maltreatment: 
impact on Medicaid. Pediatrics. 2013;132(2):312-318. 
65. 
Wang C-T, Holton J. Total estimated cost of child abuse and neglect in the United States. Chicago Il 
Prevent Child Abuse America. 2007. 
66. 
Ordway MR, Webb D, Sadler LS, Slade A. Parental reflective functioning: an approach to enhancing 
parent-child relationships in pediatric primary care. Journal of Pediatric Health Care. 2015;29(4):325-
334. 
67. 
Wolfe DA, Reppucci ND, Hart S. Child abuse prevention: Knowledge and priorities. Journal of Clinical 
Child Psychology. 1995;24(sup1):5-22. 
68. 
Kitzman H, Olds DL, Henderson CR, et al. Effect of prenatal and infancy home visitation by nurses on 
pregnancy outcomes, childhood injuries, and repeated childbearing: a randomized controlled trial. 
JAMA. 1997;278(8):644-652. 
69. 
Olds DL, Henderson Jr CR, Kitzman HJ, Eckenrode JJ, Cole RE, Tatelbaum RC. Prenatal and infancy 
home visitation by nurses: Recent findings. The Future of Children. 1999:44-65. 
70. 
Margolis PA, Stevens R, Bordley WC, et al. From concept to application: the impact of a community-
wide intervention to improve the delivery of preventive services to children. Pediatrics. 
2001;108(3):e42-e42. 
71. 
Hagen JF, Shaw JS, Duncan PM. Bright futures: Guidelines for health supervision of infants, children, 
and adolescents, Third Edition. Elk Grove Village, Il. American Academy of Pediatrics. 2008. 

72. 
Green M. Bright Futures: Guidelines for Health Supervision of Infants, Children, and Adolescents. 
Arlington, VA: National Center fo Education and Maternal and Child Health, 1994. 
73. 
Haggerty RJ, Roghmann KJ, Pless IB. Child health in the community. New York, John Wiley & Sons. 
1975: 94-116. 
74. 
Kahn RS, Wise PH, Finkelstein JA, Bernstein HH, Lowe JA, Homer CJ. The scope of unmet maternal 
health needs in pediatric settings. Pediatrics. 1999;103(3):576-581. 
75. 
Olson AL, Dietrich AJ, Prazar G, et al. Two approaches to maternal depression screening during well 
child visits. Journal of Developmental & Behavioral Pediatrics. 2005;26(3):169-176. 
76. 
Sharp L, Pantell RH, Murphy LO, Lewis CC. Psychosocial problems during child health supervision 
visits: eliciting, then what? Pediatrics. 1992;89(4):619-623. 
77. 
Horwitz SM, Storfer-Isser A, Kerker BD, et al. Barriers to the identification and management of 
psychosocial problems: changes from 2004 to 2013. Academic Pediatrics. 2015;15(6):613-620. 
78. 
Brownson RC, Colditz GA, Proctor EK. Dissemination and implementation research in health: 
translating science to practice. New York: Oxford University Press; 2012. 
79. 
Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in 
mental health services: an emerging science with conceptual, methodological, and training challenges. 
Administration and Policy in Mental Health and Mental Health Services Research. 2009;36(1):24-34. 
80. 
Chambers DA. Advancing sustainment research: challenging existing paradigms. Journal of Public 
Health Dentistry. 2011;71(suppl 1):S99-S100. 
81. 
Belsky J. Etiology of child maltreatment: A developmental€ cological analysis. Psychological Bulletin. 
1993;114(3):413. 
82. 
Brofenbrenner U, Morris PA. The ecology of developmental processes. In: Lerner R, ed. Theorectical 
models of human development. New York: Wiley; 1998:993-1028. 
83. 
Prochaska J, Diclemente C. Stages and procesess of self-change of smoking: toward an integrative 
model of change. Journal of Consulting and Clinical Psychology. 1983;51(3):390-395. 
84. 
Prochaska JO, DiClemente CC. Transtheoretical therapy: Toward a more integrative model of change. 
Psychotherapy Theory Research & Practice. 1982;19(3):276-288. 
85. 
Prochaska JO, DiClemente CC, Velicer WF, Rossi JS. Criticisms and concerns of the transtheoretical 
model in light of recent research. British Journal of Addiction. 1992;87(6):825-828. 
86. 
Williams AA, Wright KS. Engaging families through motivational interviewing. Pediatric Clinics of 
North America. 2014;61(5):907-921. 
87. 
Rubin D, Lane W, Ludwig S. Child abuse prevention. Current Opinion in Pediatrics. 2001;13(5):388-
401. 
88. 
U.S. Preventive Services Task Force 2016. Available from: 
http://www.uspreventiveservicestaskforce.org/. 
89. 
Dubowitz H, Feigelman S, Lane W, et al. Screening for depression in an urban pediatric primary care 
clinic. Pediatrics. 2007;119(3):435-443. 
90. 
Dubowitz H, Prescott L, Feigelman S, Lane W, Kim J. Screening for intimate partner violence in a 
pediatric primary care clinic. Pediatrics. 2008;121(1):e85-e91. 
91. 
Feigelman S, Dubowitz H, Lane W, et al. Screening for harsh punishment in a pediatric primary care 
clinic. Child Abuse & Neglect. 2009;33(5):269-277. 
92. 
Kim J, Dubowitz H, Hudson-Martin E, Lane W. Comparison of 3 data collection methods for gathering 
sensitive and less sensitive information. Ambulatory Pediatrics. 2008;8(4):255-260. 
93. 
Lane WG, Dubowitz H, Feigelman S, et al. Screening for parental substance abuse in pediatric primary 
care. Ambulatory Pediatrics. 2007;7(6):458-462. 
94. 
Lane WG, Dubowitz H, Feigelman S, Poole G. The effectiveness of food insecurity screening in pediatric 
primary care. International Journal of Child Health and Nutrition. 2014;3(3):130-138. 
95. 
Institute for HealthCare Improvement. 
http://www.ihi.org/resources/Pages/Tools/PlanDoStudyActWorksheet.aspx. Accessed 5/15/17. 
96. 
The NICHD Scientific Vision: The Next Decade. Vision Themes: Behavior Workshop and White Paper, 
February 17-18, 2011. https://www.nichd.nih.gov/vision/vision_themes/behavior/Pages/behavior.aspx 
Accessed September 2, 2016. 
97. 
Chamberlain P, Brown CH, Saldana L. Observational measure of implementation progress in 
community based settings: the stages of implementation completion (SIC). Implementation Science. 
2011;6(1):116. 
98. 
Saldana L, Chamberlain P, Bradford WD, Campbell M, Landsverk J. The Cost of Implementing New 
Strategies (COINS): A method for mapping implementation resources using the Stages of 
Implementation Completion. Children and Youth Services Review. 2014;39:177-182. 

99. 
Aarons GA, Ehrhart MG, Farahnak LR. The implementation leadership scale (ILS): development of a 
brief measure of unit level implementation leadership. Implementation Science. 2014;9(1):45. 
100. 
Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in 
implementation research. Administration and Policy in Mental Health and Mental Health Services 
Research. 2011;38(1):44-53. 
101. 
Straus MA, Hamby SL, Finkelhor D, Moore DW, Runyan D. Identification of child maltreatment with 
the Parent-Child Conflict Tactics Scales: Development and psychometric data for a national sample of 
American parents. Child Abuse & Neglect. 1998;22(4):249-270. 
102. 
Lane W, Dubowitz H, Frick K, Semiatin JN, Magder LS. The Safe Environment for Every Kid (SEEK) 
Program: A Cost-Effective Analysis.  139th Annual Meeting of the American Public Health Association 
Washington, D.C. 2011. 
103. 
Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice 
implementation in public service sectors. Administration and Policy in Mental Health and Mental 
Health Services Research. 2011;38(1):4-23. 
104. 
Boren SA, Balas AE. Evidence‐Based Quality Measurement. The Journal of Ambulatory Care 
Management. 1999;22(3):17-23. 
105. 
Landsverk J, Brown CH, Smith JD, et al. Design and analysis in Dissemination and implementation 
research.  (Editors)  Brownson, R., Colditz, G., Proctor, E., Oxford University Press.  Brownson RC, 
Colditz GA, Proctor EK (eds). Dissemination and Implementation Research in Health: Translating 
Science to Practice. 2nd Edition. New York: Oxford University Press; 2018 (in press). 
106. 
Aarons GA. Mental health provider attitudes toward adoption of evidence-based practice: The 
Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research. 2004;6(2):61-74. 
107. 
Dubowitz H, Black MM, Kerr MA, et al. Type and timing of mothers' victimization: effects on mothers 
and children. Pediatrics. 2001;107(4):728-735. 
108. 
Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: 
the development and validity testing of the Implementation Climate Scale (ICS). Implementation 
Science. 2014;9(1):157. 
109. 
Saldana L, Chamberlain P, Wang W, Brown CH. Predicting program start-up using the stages of 
implementation measure. Administration and Policy in Mental Health and Mental Health Services 
Research. 2012;39(6):419-425. 
110. 
Falvo DR, Smith JK. Assessing residents' behavioral science skills: Patients' views of physician–patient 
interaction. The Journal of Family Practice. 1983. 
111. 
McDonell JR, Ben-Arieh A, Melton GB. Strong Communities for Children: Results of a multi-year 
community-based initiative to protect children from harm. Child Abuse & Neglect. 2015;41:79-96. 
112. 
Aarons GA, Palinkas LA. Implementation of evidence-based practice in child welfare: Service provider 
perspectives. Administration and Policy in Mental Health and Mental Health Services Research. 
2007;34(4):411-419. 
113. 
Fitzmaurice GM, Laird NM, Ware JH. Applied Longitudinal Analysis, 2nd edition: Wiley and Sons; 
2011. 
114. 
Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the economic 
evaluation of health care programmes: Oxford University Press; 2015. 
115. 
Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qualitative Health 
Research. 2005;15(9):1277-1288. 
116. 
Miles MB, Huberman AM. Qualitative data analysis: An expanded sourcebook. 2nd ed. Thousand 
Oaks, CA: sage; 1994. 
117. 
Strauss AL, Corbin J. Basics of Qualitative Research: Techniques and Procedures for Developing 
Grounded Theory. Thousand Oaks, CA: Sage; 1998. 
118. 
Boyatzis R. Transforming qualitative information: Thematic analysis and code development. 
Thousand Oaks, CA: Sage; 1998. 
119. 
Fraser D. QSR NVivo NUD* IST Vivo Reference Guide. Melbourne: QSR International. 2000. 
120. 
Glaser BG, Strauss AL. The discovery of grounded theory: Strategies for qualitative research. New 
York: Aldine de Gruyter; 1967. 
121. 
Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation 
and variability. Field Methods. 2006;18(1):59-82. 
122. 
Patton MQ. Qualitative Evaluation and Research Methods. 4th ed. Thousand Oaks, CA: Sage; 2015.