An Implementation Strategy for the Adoption of an Evidence-Based Guideline for Pit-and-Fissure Sealants: Using a Framework From Organizational Development
Phase: NA
Conditions: Pit-and-Fissure Sealants
Sponsor: University of Pittsburgh

 
 
 An Implementation Strategy for the Adoption of an 
Evidence-Based Guideline for Pit-and-Fissure Sealants 
Protocol Number*: 17-077-E 
National Clinical Trial (NCT) Identified Number:  04682730  
Principal Investigator*: Deborah E. Polk 
Sponsor: University of Pittsburgh 
Grant Title: An implementation strategy for the adoption of an evidence-based 
pit-and-fissure sealant guideline by salaried dental providers: Using a 
framework from organizational development 
Grant Number*: U01DE027452 
Funded by: NIDCR 
Version Number:  v.0.7 
25 October 2020 

Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
ii 
 
 
Summary of Changes from Previous Version: 
 
Affected 
Section(s) 
Summary of Revisions Made 
Rationale 
 
Not applicable; this is the first version of the 
protocol 
 
Throughout 
Responding to revisions requested by Dr. Rice. 
Needed for approval by NIDCR 
Throughout 
Switching the DD sessions over to the online 
platform 
Adjusting the follow-up from 12 months to up 
to 9 months 
Adapting the study to COVID-19 physical 
distancing requirements 
Throughout 
Switching the study design from RCT to step 
wedge, from 1 year to up to 9-month follow-up 
and adding in the primary outcome sealants 
that are treatment planned but not yet placed. 
Adapting the study to fewer clinics, less 
time, and COVID-related restrictions on 
placing sealants. 
 
 
 

Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
iii 
 
CONFIDENTIALITY STATEMENT 
This document is confidential communication. Acceptance of this document constitutes agreement by 
the recipient that no unpublished information contained herein will be published or disclosed without 
prior approval of the Principal Investigator or other participating study leadership and as consistent with 
the NIH terms of award.  
 

An Implementation Strategy for the Adoption of an Evidence-Based Guideline for Pit-and-Fissure Sealants 
 
Version 0.7  
Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
iv 
 
Table of Contents 
STATEMENT OF COMPLIANCE ............................................................................................................................. 1 
INVESTIGATOR’S SIGNATURE .............................................................................................................................. 2 
1 
PROTOCOL SUMMARY ............................................................................................................................... 3 
1.1 
Synopsis ................................................................................................................................................ 3 
1.2 
Schema ................................................................................................................................................. 5 
1.3 
Schedule of Activities .......................................................................................................................... 6 
2 
INTRODUCTION ............................................................................................................................................ 7 
2.1 
Study Rationale .................................................................................................................................... 7 
2.2 
Background........................................................................................................................................... 8 
2.3 
Risk/Benefit Assessment .................................................................................................................... 9 
2.3.1 
Known Potential Risks ................................................................................................... 10 
2.3.2 
Known Potential Benefits .............................................................................................. 10 
2.3.3 
Assessment of Potential Risks and Benefits .............................................................. 10 
3 
OBJECTIVES AND ENDPOINTS .............................................................................................................. 11 
4 
STUDY DESIGN ........................................................................................................................................... 13 
4.1 
Overall Design .................................................................................................................................... 13 
4.2 
Scientific Rationale for Study Design.............................................................................................. 14 
4.3 
Justification for Intervention ............................................................................................................. 14 
4.4 
End-of-Study Definition ..................................................................................................................... 15 
5 
STUDY POPULATION ................................................................................................................................ 15 
5.1 
Inclusion Criteria ................................................................................................................................ 15 
5.2 
Exclusion Criteria ............................................................................................................................... 15 
5.3 
Lifestyle Considerations .................................................................................................................... 15 
5.4 
Screen Failures .................................................................................................................................. 16 
5.5 
Strategies for Recruitment and Retention ...................................................................................... 16 
6 
STUDY INTERVENTION(S) OR EXPERIMENTAL MANIPULATION(S) ............................................ 16 
6.1 
Study Intervention(s) or Experimental Manipulation(s) Administration ...................................... 16 
6.1.1 
Study Intervention or Experimental Manipulation Description ................................. 16 
6.1.2 
Administration and/or Dosing ....................................................................................... 17 
6.2 
Fidelity ................................................................................................................................................. 17 
6.2.1 
Interventionist Training and Tracking .......................................................................... 17 
6.3 
Measures to Minimize Bias: Randomization and Blinding ........................................................... 18 
6.4 
Study Intervention/Experimental Manipulation Adherence.......................................................... 19 
6.5 
Concomitant Therapy ........................................................................................................................ 19 
6.5.1 
Rescue Therapy ............................................................................................................. 19 
7 
STUDY INTERVENTION/EXPERIMENTAL MANIPULATION DISCONTINUATION AND 
PARTICIPANT DISCONTINUATION/WITHDRAWAL ......................................................................................... 19 
7.1 
Discontinuation of Study Intervention/Experimental Manipulation ............................................. 19 
7.2 
Participant Discontinuation/Withdrawal from the Study ............................................................... 19 
7.3 
Lost to Follow-Up ............................................................................................................................... 20 
8 
STUDY ASSESSMENTS AND PROCEDURES ..................................................................................... 20 
8.1 
Endpoint and Other Non-Safety Assessments .............................................................................. 20 
8.2 
Safety Assessments .......................................................................................................................... 24 
8.3 
Adverse Events and Serious Adverse Events ............................................................................... 24 

An Implementation Strategy for the Adoption of an Evidence-Based Guideline for Pit-and-Fissure Sealants 
 
Version 0.7  
Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
v 
 
8.3.1 
Definition of Adverse Events ........................................................................................ 24 
8.3.2 
Definition of Serious Adverse Events .......................................................................... 24 
8.3.3 
Classification of an Adverse Event .............................................................................. 25 
8.3.4 
Time Period and Frequency for Event Assessment and Follow-Up ....................... 25 
8.3.5 
Adverse Event Reporting .............................................................................................. 26 
8.3.6 
Serious Adverse Event Reporting ............................................................................... 26 
8.3.7 
Reporting Events to Participants ................................................................................. 26 
8.3.8 
Events of Special Interest ............................................................................................. 26 
8.3.9 
Reporting of Pregnancy ................................................................................................ 26 
8.4 
Unanticipated Problems .................................................................................................................... 26 
8.4.1 
Definition of Unanticipated Problems .......................................................................... 26 
8.4.2 
Unanticipated Problems Reporting .............................................................................. 27 
8.4.3 
Reporting Unanticipated Problems to Participants ................................................... 27 
9 
STATISTICAL CONSIDERATIONS .......................................................................................................... 27 
9.1 
Statistical Hypotheses ....................................................................................................................... 28 
9.2 
Sample Size Determination .............................................................................................................. 28 
9.3 
Populations for Analyses .................................................................................................................. 28 
9.4 
Statistical Analyses............................................................................................................................ 28 
9.4.1 
General Approach .......................................................................................................... 29 
9.4.2 
Analysis of the Primary Endpoint(s) ............................................................................ 29 
9.4.3 
Analysis of the Secondary Endpoint(s) ....................................................................... 29 
9.4.4 
Safety Analyses .............................................................................................................. 31 
9.4.5 
Baseline Descriptive Statistics ..................................................................................... 31 
9.4.6 
Planned Interim Analyses ............................................................................................. 31 
9.4.7 
Sub-Group Analyses ..................................................................................................... 31 
9.4.8 
Tabulation of Individual Participant Data .................................................................... 31 
9.4.9 
Exploratory Analyses ..................................................................................................... 31 
10 
SUPPORTING DOCUMENTATION AND OPERATIONAL CONSIDERATIONS .............................. 32 
10.1 
Regulatory, Ethical, and Study Oversight Considerations ........................................................... 32 
10.1.1 
Informed Consent Process ........................................................................................... 32 
10.1.2 
Study Discontinuation and Closure ............................................................................. 33 
10.1.3 
Confidentiality and Privacy ........................................................................................... 33 
10.1.4 
Future Use of Stored Specimens and Data ............................................................... 34 
10.1.5 
Key Roles and Study Governance .............................................................................. 34 
10.1.6 
Safety Oversight ............................................................................................................. 34 
10.1.7 
Clinical Monitoring .......................................................................................................... 35 
10.1.8 
Quality Assurance and Quality Control ....................................................................... 35 
10.1.9 
Data Handling and Record Keeping ............................................................................ 35 
10.1.10 
Protocol Deviations ........................................................................................................ 36 
10.1.11 
Publication and Data Sharing Policy ........................................................................... 37 
10.1.12 
Conflict of Interest Policy .............................................................................................. 37 
10.2 
Additional Considerations ................................................................................................................. 37 
10.3 
Abbreviations and Special Terms ................................................................................................... 37 
10.4 
Protocol Amendment History ........................................................................................................... 39 
11 
REFERENCES ............................................................................................................................................. 40 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
1 
STATEMENT OF COMPLIANCE 
 
The trial will be carried out in accordance with International Council on Harmonisation Good Clinical 
Practice (ICH GCP) and the following:  
 
• 
United States (US) Code of Federal Regulations (CFR) applicable to clinical studies (45 CFR 
Part 46, 21 CFR Part 50, 21 CFR Part 56, 21 CFR Part 312, and/or 21 CFR Part 812).  
 
National Institutes of Health (NIH)-funded investigators and clinical trial site staff who are 
responsible for the conduct, management, or oversight of NIH-funded clinical trials have 
completed Human Subjects Protection and ICH GCP Training. 
 
The protocol, informed consent form(s), recruitment materials, and all participant materials will 
be submitted to the IRB for review and approval. Approval of both the protocol and the consent 
form(s) must be obtained before any participant is consented. Any amendment to the protocol 
will require review and approval by the IRB before the changes are implemented to the study. 
All changes to the consent form(s) will be IRB approved; a determination will be made regarding 
whether a new consent needs to be obtained from participants who provided consent, using a 
previously approved consent form. 
 
 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
2 
INVESTIGATOR’S SIGNATURE 
The signature below constitutes the approval of this protocol and provides the necessary assurances 
that this study will be conducted according to all stipulations of the protocol, including all statements 
regarding confidentiality, and according to local legal and regulatory requirements and applicable US 
federal regulations and ICH guidelines. 
 
Principal Investigator or Clinical Site Investigator: 
Signed: 
 
Date: 
11/17/2020 
 
Name*:  Deborah Polk 
 
Title*: Assistant Professor 
 
Investigator Contact Information 
Affiliation*: University of Pittsburgh 
Address: 
Telephone: 
Email: dpolk@pitt.edu 
 
[For multi-site studies, the protocol should be signed by the clinical site investigator who is responsible 
for the day to day study implementation at his/her specific clinical site.] 
Signed: 
 
Date: 
 
 
Name:  
 
Title: 
 
 
Affiliation: 
 
 
 
 
 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
3 
1 
PROTOCOL SUMMARY 
1.1 
SYNOPSIS  
 
Title: 
An implementation strategy for the adoption of an evidence-based pit-
and-fissure sealant guideline by salaried dental providers: Using a 
framework from organizational development 
Grant Number: 
U01DE027452 
Study Description: 
The purpose of this study is to determine whether Deliberative Loops 
are effective in increasing providers’ adherence to the non-cavitated 
caries component of the American Dental Association’s pit-and-fissure 
sealant evidence-based clinical practice guideline. We use a stepped 
wedge design to randomly assign dental clinics to the Deliberative Loop 
intervention. In a Deliberative Loop, stakeholders receive background 
information, participate in a facilitated discussion, and share their views 
with leadership. The Deliberative Loop intervention is designed to help 
stakeholders form informed opinions; in this study, stakeholders will be 
forming informed opinions about the implementation interventions they 
think will increase their clinic’s adherence to the guideline. We 
hypothesize that compared with the pre-intervention period, following 
the intervention, providers will place or treatment plan sealants for 
significantly more occlusal non-cavitated carious lesions. 
Objectives*: 
 
Primary Objective: Determine whether the providers’ rates of placing or 
treatment planning sealants for occlusal NCCL increase after clinic 
stakeholders are exposed to the Deliberative Loop intervention. 
Secondary Objectives: Quantify the proportion of occlusal NCCLs that 
are treatment planned for sealants that is sealed by the end of the post-
intervention period. Estimate the total program cost, cost per clinic, and 
cost per member per month (PMPM) for the Deliberative Loop 
intervention. If effective, estimate the incremental cost-effectiveness of 
increasing sealant placements from the Deliberative Loop intervention 
compared to the pre-intervention period.    
Endpoints*: 
Primary Endpoint: The change in the providers’ rates of placing or 
treatment planning sealants for occlusal NCCLs from before to after 
clinic stakeholders are exposed to the Deliberative Loop intervention.  
 
Secondary Endpoints: Sealant treatment plans – The proportion of 
occlusal NCCLs with a treatment plan for sealants that is sealed by the 
end of the post-intervention evaluation period; Program costs – a) total 
intervention costs, cost per clinic, and costs per member per month 
(PMPM) for the DD intervention, and annualized costs for guideline 
implementation interventions and NCCL treatment costs 
Cost-effectiveness: Estimated incremental cost-effectiveness ratios 
(ICERs) for clinics after exposure to the intervention. ICERs calculated 
for annualized total costs, costs per clinic, and costs PMPM. 
Study Population: 
All persons employed by Kaiser Permanente Northwest or Permanente 
Dental Associates and who work in a Kaiser Permanente Northwest 
dental clinic at any level [e.g., part time, full time] will be eligible to 
participate in the study. This includes both service providers and front-
of-the-house staff, approximately 800 employees.  

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
4 
Phase* or Stage: 
Stage III 
Description of 
Sites/Facilities Enrolling 
Participants: 
Participating sites include the 16 general dental clinics of Kaiser 
Permanente Northwest. All clinics are located within the United States. 
Study Duration*: 
The estimated time from when the study opens to enrollment until 
completion of data collection is 12 months. 
Participant Duration: 
 It will take approximately one to two months for each individual 
participant to complete all study-related tasks, depending on how soon 
after the introductory session the Deliberative Forum can be scheduled. 
 
 
 
 
 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
5 
1.2 
SCHEMA  
Study Design for the Stepped-Wedge Study  
 
 
 
 
 
 
 
 
 
 
 
 
T = 0 months 
 
 
 
T = 1 month 
(± 2 weeks) 
 
 
 
 
T = 5 weeks 
(± 2 weeks) 
 
 
 
 
 
 
T = 2 months  
after final clinic 
meets 
(± 1 months) 
 
 
 
 
Finalize clinic enrollment (N = 16) 
Share context-setting workbook with 
clinics 
[Clinics start to roll out implementation interventions at discretion of leadership] 
Final Assessments 
Providers’ rates of placing or 
treatment planning sealants for 
occlusal NCCLs 
Randomize clinics 
Intervention 
1 Vanguard clinic plus 
5 steps each with 3 clinics 
Hold Deliberative Forum 
Administer post-meeting surveys to meeting participants 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
6 
1.3 
SCHEDULE OF ACTIVITIES  
 
 
Not applicable

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
7 
2 
 
INTRODUCTION 
 
2.1 
STUDY RATIONALE  
 
Despite the broad, evidence-based support for the use of dental sealants in permanent molars with both 
sound occlusal surfaces and noncavitated occlusal carious lesions (NCCLs) in children and adolescents, 
most general dentists are not implementing the ADA’s pit and fissure sealant guideline. Compliance 
among general dentists ranges from less than 5% to 38.5% (O'Donnell et al., 2013; Tellez, Gray, Gray, 
Lim, & Ismail, 2011). In a descriptive study of the dental practice participating in the current study (Polk 
et al., 2018), almost 95% of the general and pediatric dentists reported using sealants to prevent caries, 
but only 51% reported using sealants to arrest decay. Furthermore, using sealants to arrest decay was 
the approach least frequently adopted. Dentists reported using less effective approaches such as 
providing office remineralization (67%), prescribing a behavioral intervention (66%), indicating a “watch” 
in the chart (58%), and placing a restoration (58%). Until dental providers start implementing the 
guideline, they will continue to use less successful approaches to prevent the progression of decay from 
non-cavitation to cavitation. Thus, patients may unnecessarily develop caries that requires restoration 
and experience associated negative consequences. 
 
To date, the only implementation interventions targeting the pit and fissure sealant guideline that have 
been examined empirically involved financial incentives and education (Clarkson et al., 2008). In that 
study, the implementation strategy targeting incentives was effective in increasing implementation of 
the guideline, but the strategy targeting knowledge through education was not (Clarkson et al., 2008). 
Thus, our knowledge about how to increase adherence to this guideline is quite limited. 
 
The reason for conducting the present study is to determine whether the Deliberative Democracy 
process encourages leadership to select implementation interventions that enable clinics to increase the 
proportion of occlusal NCCLs sealed. In this intervention, stakeholders including general and pediatric 
dentists, dental hygienists, dental assistants, and clinic administrative staff will share their experiences 
and learn from the experiences of their co-workers in facilitated small group discussions. This allows 
them to discuss important issues in a richer way than our everyday interactions often enable. At the end 
of the forum, stakeholders will complete a survey in which they identify the implementation 
interventions they believe will be effective in their clinic. The results of the survey will be shared with 
KPNW Dental and clinic leadership, who will decide whether to introduce any of the recommended 
implementation interventions. We anticipate that most interventions can be implemented at the clinic 
level so that decisions will be made by clinic leaders. Some interventions may need to be implemented 
at the practice level; in these cases, decisions will be made by KPNW Dental leadership. 
 
There are different ways of achieving behavior change, including top down and bottom up. The 
Deliberative Democracy process enables leaders to make policy decisions (i.e., top down) informed by 
the opinions of those who will be affected by the policy decision (i.e., bottom up). These informed 
opinions constitute information leaders may not have access to otherwise. Thus, the process of 
participating in the Deliberative Loop will give stakeholders voice in the process. Voice has been defined 
as “expressing relevant ideas, information, and opinions about possible improvement” (Van Dyne, Ang, 
& Botero, 2003). The Deliberative Loop intervention itself is complete after the results of the survey are 
shared with leadership. The outcome of interest is the change in the providers’ rates of placing or 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
8 
treatment planning sealants for occlusal NCCLs from before to after clinic stakeholders are exposed to 
the Deliberative Loop intervention.   
 
2.2 
BACKGROUND  
We are proposing to employ a Deliberative Loop (Cavalier, 2009), which is a general term for describing 
procedures to help stakeholders develop informed opinions on issues and policy options. There are 
three steps to a Deliberative Loop. In the first step, stakeholders receive background information that 
attempts to include the full range of perspectives on the issue. In the second step, the stakeholders 
participate in facilitated small group discussions in which they share their lived experience and hear the 
lived experience of their fellow stakeholders. Achieving consensus is not a goal of the process. In the 
third step, the stakeholders complete a survey sharing their now informed opinion about the policy 
issue being addressed.  
Deliberative Loops seek to create conditions that allow a clinic to realize the principles of Deliberative 
Democracy. These principles are inclusion, reciprocity, and legitimacy. In a Deliberative Democracy, 
decision-makers solicit the informed opinion of those who will be affected by a policy (i.e., 
stakeholders). Involving those who will be affected by a policy addresses the principle of inclusion. 
Enabling those people to share their lived experiences with one another addresses the principle of 
reciprocity.  And when decision-makers use these informed opinions as a basis for determining which 
policies to enact, this establishes legitimacy. As expressed by Gutmann and Thompson (Gutmann & 
Thompson, 2004), the principles of Deliberative Democracy are realized when “citizens and officials 
[can] justify any demands for collective action by giving reasons that can be accepted by those who are 
bound by the action.” In service of the principles of Deliberative Democracy, Deliberative Loops provide 
stakeholders with the opportunity to compare values and experiences, consider a range of policy 
options, and engage relevant arguments and information (Nabatchi, Gastil, Weiksner, & Leighninger, 
2012). By bringing together stakeholders with diverse points of view, these conversations provide 
stakeholders with an opportunity to understand and consider multiple perspectives on the issue about 
which they are developing an opinion. 
Two articles review the theory and provide a critical review of Deliberative Loops in multiple countries. 
Delli Carpini and colleagues (Delli Carpini, Lomax Cook, & Jacobs, 2004) reviewed the benefits posited by 
theories of Deliberative Democracy, research (case studies, surveys, laboratory experiments, and quasi-
experimental designs) from social psychology on group deliberation, and research specifically designed 
to test the theories of Deliberative Democracy. Recently, Kuyper (Kuyper, 2018) offered a similar review 
informed by a larger body of research than was available in 2004. Both articles caution that the benefits 
of deliberation are highly context dependent. In these articles and others, researchers draw particular 
attention to how the quality of the participants’ deliberations and the quality of facilitation affect 
attainment of these benefits. A series of articles reporting on research funded by the National Institutes 
of Health and published by researchers from the University of Michigan provides guidance for assessing 
both.  These researchers detailed their particular deliberative methods for eliciting the public’s views on 
research ethics controversies (Kim, Wall, Stanczyk, & De Vries, 2009) and provided frameworks for 
assessing the quality of deliberation and facilitation (De Vries, Stanczyk, Ryan, & Kim, 2011; De Vries et 
al., 2010) derived from research on public deliberation in multiple countries (Steenbergen, Bachtiger, 
Sporndli, & Steiner, 2003; Steiner, Bachtiger, Sporndli, & Steenbergen, 2004; Stromer-Galley, 2007).  
Below is the Permanente Dental Associates guideline for dental sealants. 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
9 
Introduction   
Sealants are placed to cover pit and fissures of teeth at risk of developing caries. Current research 
supports sealants as an effective preventive service to offer patients.  
Clinical decision making  
  
  
  
  
This guideline is designed to support clinician and patient decisions about appropriate evaluation and 
treatment. It is not intended or designed as a substitute for the reasonable exercise of independent 
clinical judgment by practitioners, considering each patient's needs on an individual basis. Guideline 
recommendations apply to populations of patients. Clinical judgment and shared decision-making are 
necessary to design treatment plans for individual patients.   
 
2.3 
RISK/BENEFIT ASSESSMENT  
 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
10 
2.3.1 KNOWN POTENTIAL RISKS  
 
The only foreseeable risk to the study is informational risk (Committee on Revision to the Common Rule 
for the Protection of Human Subjects in Research in the Behavioral and Social Sciences, 2014).  Because 
the information to be obtained is neither private nor sensitive in nature, the study protocol also meets 
criteria for minimal risk per 45 CFR 46.102(i): “[T] the probability and magnitude of harm or discomfort 
anticipated in the research are not greater in and of themselves than those ordinarily encountered in 
daily life or during the performance of routine physical or psychological examinations or tests.” 
 
 
2.3.2 KNOWN POTENTIAL BENEFITS  
 
There are no known benefits of participating in the present study. If the intervention is indeed effective, 
“[D]eliberation is expected to lead to empathy with the other and a broadened sense of people’s own 
interests through an egalitarian, open-minded and reciprocal process of reasoned argumentation. 
Following from this result could be other benefits: citizens may be more enlightened about their own 
and others’ needs and experiences, may better resolve deep conflict, may be more engaged in politics, 
place their faith in the basic tenets of democracy, perceive their political system as legitimate, and lead 
a healthier civic life” (Delli Carpini et al., 2004). 
 
Research reveals a “strong link” between deliberation and the following outcomes relevant to this 
proposal (Kuyper, 2018): 
  
• 
Micro (impacts on individuals): knowledge gain, opinion and preference change. 
• 
Meso (impacts on groups and collectives): social learning, or a deeper understanding of the 
views of others; reduced polarization; meta-consensus, or a shared understanding on the nature 
of an issue and an agreement on a limited range of preferred outcomes (Dryzek & Niemeyer, 
2006). 
• 
Macro (impacts on polity): Increased perception of legitimacy of decisions as a consequence of 
informed and inclusive deliberative decision-making processes. 
 
2.3.3 ASSESSMENT OF POTENTIAL RISKS AND BENEFITS  
 
Some degree of informational risk is present in all research studies and thus cannot be avoided.  To 
minimize the risk of personal information disclosure, several measures will be taken to ensure the 
privacy and security of participant research information.  Consistent with the Privacy Rule, no member 
of the research staff will disclose participant research data to any other person or entity except as 
required by law or for authorized oversight of the research project.  The data will be used only for the 
specific purposes of this study. To protect the confidentiality of participant information, all data will be 
de-identified and labeled with a unique study ID code.  Measures to ensure the security of participant 
data are detailed in section 10.1.3. 
 
Although there are no known benefits of participation, if the intervention is effective, benefits 
consequent to dental providers’ participation may include: (1) broadening their perspectives on the use 
of pit-and-fissure sealants and by providing them with an opportunity to become more engaged in the 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
11 
establishment of guidelines for use of pit-and-fissure sealant; (2) changing professional behavior by 
encouraging more frequent use of pit-and-fissure sealants when appropriate. The knowledge to be 
gained in the present research clearly outweighs the minimal risk associated with the possible disclosure 
of information that is neither private nor sensitive. 
 
3 
OBJECTIVES AND ENDPOINTS  
 
 
OBJECTIVES 
ENDPOINTS 
JUSTIFICATION FOR ENDPOINTS 
Primary 
 
 
Determine whether the 
providers’ rates of placing or 
treatment planning sealants for 
occlusal NCCL increase after 
clinic stakeholders are exposed 
to the Deliberative Loop 
intervention. 
 
The change in the providers’ 
rates of placing or treatment 
planning sealants for occlusal 
NCCLs from before to after 
clinic stakeholders are exposed 
to the Deliberative Loop 
intervention.  
 
 
Proportion of occlusal NCCL 
with a sealant placed or 
treatment planned is an 
indicator of adherence to the 
clinical practice guideline. 
Secondary 
 
 
Sealant treatment plan 
resolution: Quantify the 
proportion of occlusal NCCLs 
that are treatment planned for 
sealants that is sealed by the end 
of the post-intervention period 
 
Program cost and cost-
effectiveness: Estimate the total 
program cost, cost per clinic, 
and cost per member per 
month (PMPM) for the 
Deliberative Loop intervention. 
If effective, estimate the 
incremental cost-effectiveness 
of increasing sealant 
placements from the 
Deliberative Loop intervention 
compared to the pre-
intervention period. 
Sealant treatment plan 
resolution: The proportion of 
occlusal NCCLs with a 
treatment plan for sealants that 
is sealed by the end of the post-
intervention evaluation period. 
 Program costs: a) total 
intervention costs, cost per 
clinic, and costs per member per 
month (PMPM) for the DD 
intervention, and annualized 
costs for guideline 
implementation interventions 
and NCCL treatment costs. 
 
Cost-effectiveness: Estimated 
incremental cost-effectiveness 
ratios (ICERs) for clinics after 
exposure to the intervention. 
ICERs calculated for annualized 
total costs, costs per clinic, and 
costs PMPM. 
 
Sealant treatment plan 
resolution: we expect dentists 
will initiate treatment plans for 
sealants if there is limited time 
availability at the time of the 
diagnosis. A treatment plan for 
a sealant will also enable a 
dental hygienist to place a 
sealant at a follow-up visit 
without additional dentist 
consultation. Because of the 
limited post-intervention 
evaluation period, we included 
treatment planned sealants in 
the primary outcome. This 
secondary aim is used to 
quantify the proportion of 
surfaces that are sealed during 
the follow-up period.   
 
An economic evaluation of a 
health system intervention is an 
important component for 
demonstrating the value of 
implementing practice change.  
This secondary aim will provide 
important data for the net 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
12 
OBJECTIVES 
ENDPOINTS 
JUSTIFICATION FOR ENDPOINTS 
financial costs for a health 
system and dental offices 
associated with the 
implementation of the DD 
intervention. Costs are assessed 
from a health system 
perspective, adjusting for 
patient-paid treatment costs.   
Tertiary/Exploratory  
 
 
E1. Determine whether 
Promotive Voice, Prohibitive 
Voice, or the perception that 
the suggestions one shares will 
be taken into consideration by 
leadership mediate the 
relationship between type of 
discussion and change in the 
proportion of NCCLs sealed. 
 
 
Promotive Voice scale (Liang, 
Farh, & Farh, 2012) (measured 
on the post-session survey 
immediately following the 
Deliberative forum) 
 
Prohibitive Voice scale (Liang et 
al., 2012) (measured on the 
post-session survey 
immediately following the 
Deliberative forum) 
 
Single item assessing self-
reported perception that the 
suggestions one shares will be 
considered by leadership 
(measured on the post-session 
survey immediately following 
the Deliberative forum) 
 
Voice is “intentionally 
expressing relevant ideas, 
information, and opinions” 
about work-related issues (Van 
Dyne, Ang, & Botero, 2003). We 
hypothesize that voice may be 
part of the pathway through 
which the experience of 
participating in the Deliberative 
Loop affects guideline 
adherence. That is, we 
hypothesize it may be part of 
the mediational pathway. 
E2. Determine whether the 
intervention was acceptable to 
stakeholders. 
 
Leadership ratings of 
acceptability and feasibility of 
resources required for the 
Deliberative Loop. (measured 
following the final Deliberative 
forum) 
 
Stakeholder ratings of 
acceptability of the Context-
Setting Workbook, group 
discussion, and process. 
(measured on the post-session 
survey immediately following 
the Deliberative forum) 
Even if an intervention is 
effective, it will not be adopted 
by the stakeholders if they do 
not find it acceptable or 
feasible. For that reason, we are 
assessing stakeholder 
perceptions of acceptability and 
feasibility. 
 
E3. Describe how the DD 
process works. 
 
Perceptions of leadership 
responsiveness in the past year 
and anticipated following the 
Deliberative forum (measured 
We are obtaining additional 
information to be able to 
provide a rich description of the 
process. 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
13 
OBJECTIVES 
ENDPOINTS 
JUSTIFICATION FOR ENDPOINTS 
on the post-session survey 
immediately following the 
Deliberative forum) 
 
Implementation interventions 
endorsed by stakeholders 
(measured on the post-session 
survey immediately following 
the Deliberative forum) 
 
Implementation interventions 
selected by leadership 
(measured in the 3 months 
following the Deliberative 
forum) 
 
4 
STUDY DESIGN 
 
4.1 
OVERALL DESIGN 
We hypothesize that compared with the pre-intervention period, following the intervention, providers 
will place or treatment plan sealants for significantly more occlusal non-cavitated carious lesions. 
 
This is a stepped wedge randomized trial design. 
 
Every clinic will eventually experience the intervention. Clinics will be randomized to one of five 
intervention periods. We will randomize three clinics at a time. There is a six-week window in which to 
run the three clinics that are in a given step and then a week gap before the next step's six-week 
window starts. 
 
There will be an intervention period and a non-intervention period that will determine when the clinic 
experiences the Deliberative Loop (DL) process. The cumulative duration of the study intervention is 105 
minutes. This includes a 15-minute overview in the month prior to the DL session and 90 minutes 
dedicated to the DD session and post session survey. In the Overview, participants will receive the 
Context-Setting Workbook. In the DL session, participants will engage in the Deliberative forum and 
complete the post-forum survey identifying implementation interventions they believe will or will not be 
effective in their own clinic. The final endpoint assessment for each clinic will occur 2 months after the 
final clinic’s Deliberative forum. 
 
The economic evaluation has two components: a program cost analysis and an incremental cost-
effectiveness analysis (CEA). A program cost analysis (PCA) (Drummond, Sculpher, Claxton, Stoddart, & 
Torrance, 2015) will be conducted to quantify the total direct costs (labor time and non-labor resources) 
of developing and implementing the Deliberative Loop intervention, including the annualized costs of 
selected clinic implementation interventions. For this study, we will use PCA to value resources using an 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
14 
opportunity cost method, i.e., that resources used for the intervention cannot be used for other 
purposes, from a health system perspective.  
 
To characterize the DD process, the post-session survey will also include questions addressing the 
stakeholders’ perceptions of leadership responsiveness to feedback in the past year, barriers they 
identify, perceptions of promotive and prohibitive voice, anticipated responsiveness to the feedback 
from the forum, perceptions of helpfulness of the Context-Setting Workbook, perceptions of helpfulness 
of the Deliberative forum, and perceptions of helpfulness to the stakeholder of the forum. We will also 
assess leadership’s perceptions of the feasibility of the Deliberative Loop process. Finally, we will use 
surveys and conduct qualitative interviews with clinic managers to obtain information about which 
implementation interventions were actually implemented in their clinics. 
 
This will be a single-site, multi-clinic (n = 16) trial. 
 
We are proposing to employ a Deliberative Loop (Cavalier, 2009), which is a general term for describing 
procedures to help stakeholders develop an informed opinion on issues and policy options. Key 
elements of Deliberative Loops include providing stakeholders the following elements: 
• 
background information on an issue 
• 
a facilitated conversation as a member of a group of stakeholders that offers a range of 
perspectives on an issue as a result of its members’ diverse lived experience 
• 
the opportunity to record their views after having become informed with relevant facts, expert 
information, and an understanding of the effects issues and policy options can have on diverse 
members of a community (Cavalier, 2009).  
 
Clinics in the non-intervention period will experience standard processes for decision-making and 
administrative support. Selection of the timing of the non-intervention period for clinics will be based on 
randomization. 
 
Interim analyses will not be used to guide the current research study. 
 
Given the limited number of clinics, we will not be stratifying. 
 
We are not including any additional sub-studies in this protocol. 
 
4.2 
SCIENTIFIC RATIONALE FOR STUDY DESIGN 
 
Given the almost complete lack of prior research testing interventions to improve clinics’ 
implementation of the ADA’s pit-and-fissure sealant guideline for occlusal NCCLs, we selected a stepped 
wedge clinical trial. Every clinic will eventually receive the intervention, which we think will be 
beneficial. A stepped wedge design also likely provides more power and accounts for intraclass 
correlation in a more efficient way than clustered randomized trials. By having non-intervention and 
intervention periods, we will be able to establish a benchmark for provider behavior (i.e., sealing NCCLs), 
which could drift over the study period or change in response to factors other than the intervention. 
 
4.3 
JUSTIFICATION FOR INTERVENTION 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
15 
 
The number and frequency of the intervention contacts is typical for a Deliberative Loop. The length of 
time of these intervention contacts is shorter than is typical due to the constraints of administering the 
intervention in a busy clinic setting. The minimum-acceptable participation in, or exposure to, the 
intervention to have evaluable data requires participation in two of the three elements of the 
intervention: reading the Context-Setting Workbook and completing a post-session survey.  
 
4.4 
END-OF-STUDY DEFINITION 
 
The end of the study, defined as when we will stop pulling from the EHR the proportion of NCCLs sealed, 
will occur 2 months after the last clinic completes their Deliberative Forum. 
5 
STUDY POPULATION 
 
5.1 
INCLUSION CRITERIA 
 
All persons employed by Kaiser Permanente Northwest or Permanent Dental Associates and who work 
in a Kaiser Permanente Northwest dental clinic at any level [e.g., part time, full time] will be eligible to 
participate in the study. This includes both service providers and front-of-the-house staff. We assume all 
participants are at least 18 years old. 
 
Please note: To measure the provider behavior that is our primary outcome, we are using patient data 
from the Electronic Health Record. Patients are not our participants, however. We describe use of the 
EHR data in section 8.1. Per CHR policy, clinical data for patients will be excluded from the analysis for 
patients who have requested their data not be used for research purposes. 
 
The study includes a pragmatic evaluation of clinical information collected in the routine conduct of 
patient dental care. Data collected electronically as part of care will be collected for all patients ages 6-
80 with a dental office visit during the clinic evaluation period and who have a documented NCCL (or 
suspected NCCL) on an occlusal tooth surface. These patients will not be recruited directly or contacted 
by study or dental office staff to obtain patient-reported data.  
 
Note: We obtained a waiver of signed consent to access and utilize these data. Patient data will be 
stored securely using established procedures at CHR, and data sharing among the study team will be 
governed by an appropriate data use and data transfer agreements. 
 
5.2 
EXCLUSION CRITERIA 
 
There are no exclusions for KPNW and PDA staff who are otherwise eligible for the intervention.  
 
5.3 
LIFESTYLE CONSIDERATIONS 
 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
16 
N/A 
5.4 
SCREEN FAILURES 
 
N/A  
5.5 
STRATEGIES FOR RECRUITMENT AND RETENTION 
 
Clinic managers and leadership will reach out to potential participants. We will leave the exact method 
to the discretion of the managers and leaders. 
Participants will not receive compensation, nor will they be provided any incentives for study 
participation. 
 
6 
STUDY INTERVENTION(S) OR EXPERIMENTAL MANIPULATION(S) 
 
6.1 
STUDY INTERVENTION(S) OR EXPERIMENTAL MANIPULATION(S) ADMINISTRATION 
 
6.1.1 STUDY INTERVENTION OR EXPERIMENTAL MANIPULATION DESCRIPTION 
 
Presentation of the Context-Setting Workbook 
Background information will be given to the stakeholders before they engage in the Deliberative Forum. 
One goal of these Context-Setting Workbook is to present the diverse perspectives held by the 
stakeholders regarding barriers they face in adhering to the guideline. Stakeholders should see their 
own perspective or voice in the materials. To accomplish this goal, we surveyed the stakeholders about 
their views regarding barriers. We will also provide information about a range of possible 
implementation interventions together with their targets. All implementation interventions included in 
the Context-Setting Workbook will have been vetted by KPNW leadership to ensure that they are open 
for consideration. 
 
Exposure to DD Session 
Stakeholders from a given clinic will come together for a Deliberative Forum (i.e., facilitated 
conversation), enabling them to hear the full range of perspectives held by their fellow colleagues and 
stakeholders and to share their own perspective regarding how best to implement the guideline in 
small-group discussions. 
 
Completion of Post-intervention Survey 
Following the Deliberative Forum, each stakeholder will have the opportunity to record their opinions 
regarding possible implementation interventions. These opinions will be shared with the clinic’s 
decision-makers to help guide their decision-making about which implementation interventions to 
deploy. If a stakeholder happens to be absent the day of their clinic’s DD session, they will still have the 
opportunity to complete the survey. 
 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
17 
The mechanistic targets of the intervention are employees’ perceptions of promotive voice, prohibitive 
voice, and anticipation that KP leadership will take their recommendations under consideration.  
 
The targeted endpoint is the change in the providers’ rates of placing or treatment planning sealants for 
occlusal NCCLs from before to after clinic stakeholders are exposed to the Deliberative Loop 
intervention. 
 
All clinics will have a non-intervention period and an intervention period.  
6.1.2 ADMINISTRATION AND/OR DOSING 
 
Introduction to the Study & Presentation of Context-Setting Workbook 
At a clinic meeting one month before the clinic’s Deliberative forum, one member of the study staff will 
give a 15-minute presentation providing background information about the DISGO study and the 
Deliberative forum. In addition, stakeholders (i.e., general and pediatric dentists, dental hygienists, 
Expanded Function Dental Assistants (EFDAs), dental assistants, and front-of-the-house staff) will watch 
a five minute video demonstrating the online platform and be given the Context-Setting Workbook. We 
will ask the stakeholders to review the Context-Setting Workbook before participating in their clinic’s 
Deliberative forum. 
 
Exposure to DD Session 
Stakeholders working in intervention clinics will participate in clinic-specific Deliberative forums. Given 
KPNW Dental policy, we expect that all stakeholders who are present on the day of the forum will 
attend and participate in the forum. These forums will take place via an online platform. The forum will 
start with a quick review of the material in the Context-Setting Workbook. Then, depending on the 
number of stakeholders in the clinic, the stakeholders will be divided into small groups of 5 – 8 members 
with one facilitator per group. The membership within each group will be representative of the diversity 
among job categories (i.e., general and pediatric dentists, dental hygienists, EFDAs, dental assistants, 
and front-of-the-house staff). The facilitators will lead the small groups in focused discussions of barriers 
and implementation interventions to address those barriers. The facilitators will seek to elicit from the 
stakeholders their lived experience and the reasoning for their opinions but will discourage any pressure 
to arrive at consensus. The Deliberative forums will last for one hour.  
 
Completion of Post-intervention Survey  
Immediately following the Deliberative forum, stakeholders will complete a survey on which they will 
record their opinions about the implementation interventions they recommend their clinic adopt. If a 
stakeholder is absent on the day of their forum, we will still give them the post-session survey to 
complete. The results of the surveys will be shared with whatever level of leadership has the ability to 
enact the recommendations. Thus, for implementation interventions that require system-wide 
intervention, such as continuing education, those recommendations will be shared with the Clinical 
Effectiveness Council. For implementation interventions that require intervention at the clinic level, 
those recommendations will be shared with clinic leadership and administration. 
 
6.2 
FIDELITY 
 
6.2.1 INTERVENTIONIST TRAINING AND TRACKING 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
18 
 
Facilitators will be drawn from persons working as facilitators who work with Common Ground for 
Action. The role of the facilitator is to make sure that every stakeholder has the opportunity to share 
their lived experience. In addition, the facilitators encourage stakeholders to support their views with 
specific reasons. They will receive training in the form of facilitation specific to Deliberative Loop forums 
from Tim Dawson, a study consultant. All Deliberative forums will be recorded. Following each 
Deliberative forum, we will code the recordings to assess how well the facilitator adhered to the 
protocol and share feedback with the facilitators. The following elements will be assessed: 
1. Impartial/non-judgmental: Avoids sharing their own opinion or appraising the opinion of 
participants. 
2. Task-oriented: keeps participants on track if discussion gets side-tracked. Reminds participants 
of goals and issues that are central to discussion. 
3. Elicits viewpoints from every participant/ensures that no one dominates the discussion: 
Encourages equal participation of each participant, makes room for participants to share their 
opinions. 
4. Conflict resolution: resolves confrontation and maintains focus on issues rather than 
personalities 
Immediately following the Deliberative forum, we will administer a survey to the participating 
stakeholders. On that survey, we will include two items assessing fidelity (e.g., “Today’s discussion 
caused me to consider points of view that I had not previously considered”) scored on a five-point 
Likert-type scale ranging from “Strongly Disagree” to “Strongly Agree” together with a free response box 
for comments and suggestions. We will use descriptive statistics to analyze these data. If we determine 
from the fidelity assessment that the training for the facilitators could be improved, we will make that 
change to the training. 
 
We will take several measures to quantify the degree to which the Deliberative forums occurred as 
planned. We will assess which stakeholders from each clinic attended. We will code the recordings to 
assess whether stakeholders participated in the facilitated discussions. We will record whether 
stakeholders returned a survey at the end of the forum.  
 
6.3 
MEASURES TO MINIMIZE BIAS: RANDOMIZATION AND BLINDING 
 
The study will not be blinded. One clinic will be designated as a vanguard clinic for the intervention. The 
vanguard clinic will be randomly selected as follows: the 16 clinics will be ranked by number of 
providers. The sample function in R will be used to randomly select one clinic from the 8 clinics in the 
middle 50% This will be our vanguard clinic. This is done to ensure the vanguard clinic is generally 
representative of most clinics in terms of number of providers.  
 
The remaining fifteen clinics will be randomly assigned to one of five intervention periods. We will use a 
restricted randomization approach to ensure relative balance in providers randomized in each period. 
We will rank the fifteen clinics by provider size and divide them into five groups of three. We will 
randomly select one clinic from each group for each intervention period. We will use the sample() 
function in R to randomly assign clinics into intervention periods, with three clinics assigned during each 
intervention period. The sample() function will be used to randomize each group of three into a step. 
 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
19 
6.4 
STUDY INTERVENTION/EXPERIMENTAL MANIPULATION ADHERENCE 
There are three parts to the intervention: The Context-Setting Workbook; the Deliberative Loop; and the 
post-session survey.  
 
To measure stakeholders’ engagement with the Context-Setting Workbook, we will ask them on 
the post-session survey whether they read the Context-Setting Workbook.  
 
To measure stakeholders’ engagement in the Deliberative Loop, we will take attendance, to see 
whether they attended. We will also review the transcripts to determine whether they 
participated.  
 
To measure stakeholders’ exposure to the post-session survey, we will have them put their study 
ID on the survey. 
 
6.5 
CONCOMITANT THERAPY 
 
N/A 
6.5.1 RESCUE THERAPY 
 
N/A 
 
7 
STUDY INTERVENTION/EXPERIMENTAL MANIPULATION DISCONTINUATION AND 
PARTICIPANT DISCONTINUATION/WITHDRAWAL 
 
7.1 
DISCONTINUATION OF STUDY INTERVENTION/EXPERIMENTAL MANIPULATION 
 
Not applicable. 
 
7.2 
PARTICIPANT DISCONTINUATION/WITHDRAWAL FROM THE STUDY 
We will discontinue participants from the study if they cease to be employed by either PDA or KPNW. 
The reason for participant discontinuation or withdrawal from the study will be recorded on the 
enrollment tracker. Participants who undergo the consent process, receive the study intervention, and 
subsequently withdraw or are discontinued from the study will not be replaced. 
 
Because the study intervention process is fundamentally a group process, we may discontinue 
participants from the study intervention if the participants disrupt the process for other participants. 
Disruption includes threatening physically. Disagreeing is not included in the definition of disruption. 
When we discontinue a participant from the study intervention but not the study, we will give the 
participant the opportunity to complete remaining study procedures. When we discontinue a 
participant from the study, we will collect the reason(s) we identified for discontinuing the participant 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
20 
from the intervention and the methods we used to determine the need to discontinue the participant. 
Given the one-time-only nature of the Deliberative loop for each clinic, we are unable to anticipate a 
situation in which we would discontinue a participant from the study intervention temporarily. 
 
Clinic staff from dental offices receiving the DD intervention may relocate to a clinic still in its non-
intervention period during the evaluation period. To minimize contamination associated with staff 
movement between clinics, we will monitor provider staffing changes and NCCL treatment decisions 
(procedures, treatment plans, etc.) of providers who change clinic locations. We will assess the potential 
for staff change on study results and consider excluding data for these providers if warranted. We 
expect any provider cross-contamination would reduce the differences between non-intervention and 
intervention periods, thus making the results more conservative than if no provider movement had 
occurred. 
 
For participants who have been discontinued from the study intervention but remain in the study for 
follow-up, we will continue contact to be able to capture adverse events (AE), serious adverse events 
(SAE), and unanticipated problems (UPs). 
7.3 
LOST TO FOLLOW-UP 
Participants will be considered lost to follow-up if they cease to be employed by either PDA or KPNW. If 
they remain employed but don’t attend the DL session or don’t complete the post-session survey, they 
will be included in the analyses (see 9.3 Populations for Analyses). 
8 
STUDY ASSESSMENTS AND PROCEDURES 
 
8.1 
ENDPOINT AND OTHER NON-SAFETY ASSESSMENTS 
 
Primary Objective 
The primary objective of this study is to determine whether following the initiation of the Deliberative 
Loop, sealant placement rates on occlusal tooth surfaces with a non-cavitated carious lesion (NCCL) 
increase significantly during the post-initiation period compared to sealant placement on NCCLs during 
the pre-intervention period. To fulfill the primary objective, we will assess at both the provider and clinic 
levels the change in the providers’ rates of placing or treatment planning sealants for occlusal NCCLs 
from before to after clinic stakeholders are exposed to the Deliberative Loop intervention. 
((sealant+TPS)/NCCL)OT1 –  ((sealant+TPS)/NCCL)OT0  
where TPS is treatment planned sealant and OT0 and OT1 refer to all occlusal (O) surfaces for time 
periods T0 (before) and T1 (after) the intervention. Currently, similar to most practicing dentists, PDA 
dentists do not use diagnosis codes. Instead, dentists use clinical observations and radiographs to 
establish clinical finding of caries by tooth and surface, and estimate the pre-intervention depth (E1, E2, 
D1, etc.) of the affected surface. Practitioners then enter these data in the clinical findings tab of the 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
21 
electronic records by tooth surface.  Study staff will draw these data from the electronic health record 
(i.e., HealthConnect [EPIC CARE] WISDOM module) to define eligible tooth surfaces with an E1 or E2 
caries finding and use procedure data to determine how the tooth surface was treated.  
Definitions from the HealthConnect [EPIC CARE] WISDOM module: 
An NCCL is defined as an unrestored primary or permanent tooth surface of a molar or premolar 
with a documented clinical finding of caries at depths E1 or E2 (NCCL, non-cavitated, or incipient 
caries). Primary molars are included in the PDA guideline, which will support analyses for young 
children, adolescents and adults. Eligible teeth with multi-surface carious lesions will be 
excluded if the non-occlusal surface(s) are frank caries (depth: D1-D3), which would likely 
warrant surgical restoration. 
A sealant is defined as CTD procedure code D1351 (Sealant per tooth) or D1353 (Sealant repair 
per tooth).  
A treatment planned sealant is defined as the presence of a sealant placement, and date 
corresponding to the NCCL, documented visit in the Treatment Plan table in HealthConnect.   
Other non-surgical treatment procedures include D1352 (PRR), D1354 (interim caries arresting 
medicament, D1999 (unspecified preventive procedure, D1208 (topical fluoride), and D1206 
(fluoride varnish).  
Occlusal surfaces are denoted (O) in HealthConnect for primary and permanent teeth.  
Secondary Objectives 
A secondary objective is to quantify the proportion of occlusal NCCLs that are treatment planned for 
sealants that are sealed during the study period. Details of the analysis plan are provided in section 9.4.3 
below 
Another secondary objective is to estimate the total program cost, cost per clinic, and cost per member 
per month (PMPM) for the Deliberative Loop intervention. The cost-effectiveness analysis will be 
focused on the cost of developing and conducting the DD intervention (using an implementation costing 
approach that excludes the research-related costs that would not accrue to health plans doing this on 
their own), the cost of selected implementation interventions following the DD session, and the clinical 
service delivery costs for eligible tooth surfaces during the evaluation period. Details of the analysis plan 
are provided in section 9.4.3 below. 
Exploratory Objectives 
Acceptability & Feasibility 
Following the final deliberative forum, we will assess KPNW Dental and PDA management’s perceptions 
of acceptability and feasibility of the Deliberative Loop process. We will measure the resources required 
to conduct the deliberative forums including the researchers’ time preparing the Context-Setting 
Workbook and preparing for the deliberative forums. We will use descriptive statistics to quantify the 
resources. We will then share this information with the clinic, KPNW Dental, and PDA management 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
22 
teams and have them rate the acceptability and feasibility of the deliberative loop on a 5-point Likert 
scale.  
Immediately following the deliberative forum on the post-session survey, we will obtain measures of 
acceptability from the stakeholders.  
Item 
3-item scale assessing helpfulness of the Context-Setting Workbook to the discussion 
3-item scale assessing helpfulness of the discussion to identifying implementation interventions 
3 items assessing experience during the discussion 
We will assess the acceptability of the Context-Setting Workbook via three items (e.g., “I found the 
written materials clear and easy to understand”) scored on a five-point Likert-type scale ranging from 
“Strongly Disagree” to “Strongly Agree” together with a free response box for comments and 
suggestions.  
We will assess the helpfulness of the group discussions via three items (e.g., “I found the group 
discussions informative”) scored on a five-point Likert-type scale ranging from “Strongly Disagree” to 
“Strongly Agree” together with a free response box for comments and suggestions.  
We will assess the acceptability of the process via three items (e.g., “Today’s discussion made me more 
likely to become engaged in my clinic’s efforts to resolve issues related to implementing the incipient 
caries guideline” and “In the future, I would be willing to participate in deliberative forums like the one 
today”) scored on a five-point Likert-type scale ranging from “Strongly Disagree” to “Strongly Agree” 
together with a free response box for comments and suggestions.  
Examining the mediating role of Voice 
We will examine whether Voice mediates the relationship between type of discussion (i.e., 
preintervention Unit-based team discussion and Deliberative Loop discussion) and the primary outcome 
(i.e., the change in the providers’ rates of placing or treatment planning sealants for occlusal NCCLs from 
before to after clinic stakeholders are exposed to the Deliberative Loop intervention.). 
Item 
5-item scale assessing Promotive Voice 
5-item scale assessing Prohibitive Voice 
1-item scale assessing perceptions of future leadership responsiveness 
1 item assessing overall perception of Voice 
Different aspects of Voice will be measured in different ways. Measures of voice distinguish between 
expressing new ideas or suggestions for improving the clinic, which is called promotive voice, and 
expressing concern about work practices that may be harmful to the clinic, which is called prohibitive 
voice (Liang, Farh, & Farh, 2012). Promotive voice focuses on realizing ideals or possibilities; whereas 
prohibitive voice focuses on stopping or preventing harm. The perception of having the opportunity to 
share one’s views will be measured using the Promotive and Prohibitive Voice scales (Liang et al., 2012).  

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
23 
For both types of discussion (see above), both Promotive Voice and Prohibitive Voice will be assessed 
immediately following the DL forum, via a survey administered to all stakeholders attending the meeting 
(i.e., post-session survey). Each scale has five items and is scored on a five-point Likert-type scale ranging 
from “Strongly Disagree” to “Strongly Agree.” Six-week test-retest reliability was shown to be 0.42 for 
Promotive Voice and 0.44 for Prohibitive Voice. In validation studies, these scales were shown to have 
convergent and discriminant validity. Confirmatory analysis that also included measures of psychological 
safety, felt obligation for constructive change, and organization-based self-esteem, demonstrated that a 
five-factor model in which each scale loaded on a separate factor fit best. 
Another aspect of Voice is the perception that the suggestions that one shares will be taken into 
consideration by leadership. To measure this perception, we are administering a single item “I am 
confident KP leadership (Upper Management) will take our suggestions into consideration.” This item 
will be assessed immediately following the DL forum, via a survey administered to all stakeholders 
attending the meeting (i.e., post-session survey). It is scored on the same scale as the Promotive and 
Prohibitive Voice scales. Finally, we will measure an overall assessment of voice using a single item 
“Today’s discussion made me feel as though my voice has been heard.” This item will be assessed 
immediately following the DL forum via a paper survey administered to all stakeholders attending the 
meeting (i.e., post-session survey). It is scored on the same scale as the Promotive and Prohibitive Voice 
scales. 
Characterizing the DD Process 
We will also conduct exploratory analyses to learn more about the DD process.  
Item 
For each barrier, items assessing endorsement of implementation interventions 
2 items assessing perceptions of past leadership responsiveness 
3 items assessing perceived barriers 
2 items assessing fidelity 
The usual product of the intervention is the stakeholders’ informed opinions about what 
implementation interventions their clinic should adopt. Immediately following the deliberative forum, 
on the post-session survey, for each barrier, stakeholders will indicate the degree to which they agree 
using a five-point scale (strongly disagree to strongly agree) that their clinic should adopt a series of 
implementation interventions. In addition to the list of implementation interventions, stakeholders will 
be able to write in additional interventions not included in the list or indicate that no interventions are 
needed. They will also be able to select interventions for barriers not included in the Deliberative Forum 
agenda and Context-Setting Workbook. 
Immediately following the Deliberative forum, we will assess stakeholders’ perceptions of leadership 
responsiveness in the past year with two questions, which are scored on a five-point Likert-type scale 
ranging from “Strongly Disagree” to “Strongly Agree” together with free response boxes for comments 
and suggestions. 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
24 
Immediately following the Deliberative forum, we will assess the barriers stakeholders perceive in their 
clinic (3 items; e.g., “In my clinic, the following are barriers: Concern that sealants make things worse”). 
These items will be scored on a five-point Likert-type scale ranging from “Strongly Disagree” to “Strongly 
Agree” together with free response boxes for comments and suggestions.  
Immediately following the Deliberative Forum, we will measure stakeholders’ perceptions about the 
discussions using the following two items: “Today’s discussion gave me an understanding of important 
issues related to implementing the incipient caries guideline;” and “Today’s discussion caused me to 
consider points of view that I had not previously considered.” Both items are scored on a five-point 
Likert-type scale ranging from “Strongly Disagree” to “Strongly Agree” together with free response 
boxes for comments and suggestions. We are using these items to assess fidelity,. 
To assess the implementation interventions adopted by the clinics, we will take several steps. First, we 
will provide a template to clinics that they can fill out to document progress of the implementation 
process for 3 months. We will collect copies of the progress sheet quarterly. After 3 months, select clinic 
staff will be invited for qualitative interviews and the progress sheets will serve as the basis for the 
interview to better understand the implementation process in the intervention clinics. 
8.2 
SAFETY ASSESSMENTS 
 
N/A 
 
8.3 
ADVERSE EVENTS AND SERIOUS ADVERSE EVENTS 
 
8.3.1 DEFINITION OF ADVERSE EVENTS 
 
An adverse event is any untoward or unfavorable medical occurrence in a human subject, including any 
abnormal sign (for example, abnormal physical exam or laboratory finding), symptom, or disease, 
temporally associated with the subject’s participation in the research, whether or not considered 
related to the subject’s participation in the research. 
 
8.3.2 DEFINITION OF SERIOUS ADVERSE EVENTS 
 
A serious adverse event (SAE) is one that meets one or more of the following criteria: 
• 
Results in death 
• 
Is life-threatening (places the subject at immediate risk of death from the event as it occurred) 
• 
Results in inpatient hospitalization or prolongation of existing hospitalization 
• 
Results in a persistent or significant disability or incapacity 
• 
Results in a congenital anomaly or birth defect   
An important medical event that may not result in death, be life threatening, or require hospitalization 
may be considered an SAE when, based upon appropriate medical judgment, the event may jeopardize 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
25 
the subject and may require medical or surgical intervention to prevent one of the outcomes listed in 
this definition. 
 
8.3.3 CLASSIFICATION OF AN ADVERSE EVENT 
8.3.3.1 SEVERITY OF EVENT 
For adverse events (AEs) not included in the protocol defined grading system, the following guidelines 
will be used to describe severity.  
 
• 
Mild – Events result in only mild, momentary psychological or physical discomfort that 
dissipates quickly and that can be resolved without outside intervention.  
• 
Moderate – Events result in more prolonged and intense psychological or physical distress that 
may require informal outside intervention.  
• 
Severe – Events result in significant psychological or physical distress that extends beyond the 
duration of a single focus group meeting and/or is associated with damage or destruction of 
physical property and/or necessitates the intervention of medical, local security, or police 
personnel. 
 
8.3.3.2 RELATIONSHIP TO STUDY INTERVENTION/EXPERIMENTAL MANIPULATION 
 
All adverse events (AEs) will have their relationship to study procedures, including the intervention, 
assessed by appropriately trained study personnel.  The degree of certainty about causality will be 
graded using the categories below.  
 
• 
Related – The AE is known to occur with the study procedures, there is a reasonable possibility 
that the study procedures caused the AE, or there is a temporal relationship between the study 
procedures and the event. Reasonable possibility means that there is evidence to suggest a 
causal relationship between the study procedures and the AE. 
• 
Not Related – There is not a reasonable possibility that the study procedures caused the event, 
there is no temporal relationship between the study procedures and event onset, or an 
alternate etiology has been established. 
 
8.3.3.3 EXPECTEDNESS  
There are no anticipated adverse effects associated with this protocol. 
 
8.3.4 TIME PERIOD AND FREQUENCY FOR EVENT ASSESSMENT AND FOLLOW-UP 
 
The occurrence of an adverse event (AE) or serious adverse event (SAE) may come to the attention of 
study personnel (i.e., discussion group facilitator) during discussion group meetings.  As no probing 
questions will be asked by the group facilitator, all AEs and SAEs will be treated as unsolicited events.  
Documentation and reporting of AEs and SAEs will occur throughout the duration of the study as 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
26 
appropriate.  If the AE or SAE is not resolved within the time period of the group meeting, subsequent 
meetings will be monitored for reoccurrence or escalation. 
8.3.5 ADVERSE EVENT REPORTING  
 
All AEs, not otherwise precluded per the protocol, will be documented by study personnel present at the 
time of occurrence.  Documentation will include the time and date of the event, a brief description of 
the event, an assessment of severity, identification of the involved parties, and whether and how the 
issue was resolved.  The event will be reported to the principal investigator within a reasonable 
timeframe, depending on the event’s severity.  Following report of the event, the principal investigator, 
site investigator, and the individual who recorded the event will evaluate safety events for relatedness 
and seriousness to the DD study intervention and discuss the necessity of taking corrective and/or 
preventive action.   
8.3.6 SERIOUS ADVERSE EVENT REPORTING  
 
In consultation with the PI, a trained member of the study team will be responsible for conducting an 
evaluation of a serious adverse event and shall report the results of such evaluation to the reviewing 
Institutional Review Board (IRB) as soon as possible, but in no event later than 5 working days after the 
investigator first learns of the event. 
 
 
8.3.7 REPORTING EVENTS TO PARTICIPANTS  
 
N/A 
 
8.3.8 EVENTS OF SPECIAL INTEREST  
 
N/A 
8.3.9 REPORTING OF PREGNANCY  
 
N/A 
 
8.4 
UNANTICIPATED PROBLEMS 
 
8.4.1 DEFINITION OF UNANTICIPATED PROBLEMS 
 
This protocol uses the definition of Unanticipated Problems as defined by the Office for Human 
Research Protections (OHRP).  OHRP considers unanticipated problems involving risks to participants or 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
27 
others to include, in general, any incident, experience, or outcome that meets all of the following 
criteria: 
 
• 
Unexpected in terms of nature, severity, or frequency given (a) the research procedures that are 
described in the protocol-related documents, such as the Institutional Review Board (IRB)-
approved research protocol and informed consent document; and (b) the characteristics of the 
participant population being studied; 
• 
Related or possibly related to participation in the research (“possibly related” means there is a 
reasonable possibility that the incident, experience, or outcome may have been caused by the 
procedures involved in the research); and 
• 
Suggests that the research places participants or others at a greater risk of harm (including 
physical, psychological, economic, or social harm) than was previously known or recognized. 
 
8.4.2  UNANTICIPATED PROBLEMS REPORTING  
 
The investigator will report unanticipated problems (UPs) to the reviewing Institutional Review Board 
(IRB) and the lead principal investigator (PI). The UP report will include the following information: 
 
• 
Protocol identifying information: protocol title and number, PI’s name, and the IRB project 
number 
• 
A detailed description of the event, incident, experience, or outcome  
• 
An explanation of the basis for determining that the event, incident, experience, or outcome 
represents an UP 
• 
A description of any changes to the protocol or other corrective actions that have been taken or 
are proposed in response to the UP 
 
To satisfy the requirement for prompt reporting, UPs will be reported using the following timeline:   
 
• 
UPs that are serious adverse events (SAEs) will be reported to the IRB, the lead PI, and to the 
study sponsor (University of Pittsburgh) within 5 days of the investigator becoming aware of the 
event  
• 
Any other UP will be reported to the IRB, the lead PI, and to the study sponsor within one week 
of the investigator becoming aware of the problem  
• 
All UPs should be reported to appropriate institutional officials (as required by an institution’s 
written reporting procedures), the supporting agency head (or designee), and the Office for 
Human Research Protections (OHRP) within 30 days of the IRB’s receipt of the report of the 
problem from the investigator] 
 
8.4.3 REPORTING UNANTICIPATED PROBLEMS TO PARTICIPANTS  
 
N/A 
9 
STATISTICAL CONSIDERATIONS  
 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
28 
9.1 
STATISTICAL HYPOTHESES 
 
• 
Primary Endpoint:  
We hypothesize that compared with the pre-intervention period, following the intervention, 
providers will place or treatment plan sealants for significantly more of the occlusal non-cavitated 
carious lesions. Alternatively, our null hypothesis is that there will be no difference in the effects of 
Deliberative Democracy on guideline implementation. 
 
 
• 
Secondary Endpoints: 
We hypothesize that the Deliberative Loop will be cost effective at 6 months following participation 
in the Deliberative Loop forums.  
 
9.2 
SAMPLE SIZE DETERMINATION 
 
The primary outcome is the change in the providers’ rates of placing or treatment planning sealants for 
occlusal NCCLs from before to after clinic stakeholders are exposed to the Deliberative Loop 
intervention. Fifteen clinics will be randomized in a stepped wedge design. Three clinics will be 
randomized during each of five time periods. Our sample size will be the number of clinics participating 
in the trial, as well as the number of providers within those clinics. 
 
Our null hypothesis is that the change in providers’ rates of placing or treatment planning sealants for 
occlusal NCCLs is not different between the non-intervention and intervention periods. Our alternative 
hypothesis is that the change in providers’ rates of placing or treatment planning sealants for occlusal 
NCCLs is different between the non-intervention and intervention periods.  We will assume a Type I 
error of 0.05. Randomizing 15 clinics along with 1 vanguard clinic, using a stepped wedge approach, we 
will have at least 80% power, and close to 100% under several scenarios. This assumes the non-
intervention period will see a 3% rate of placing or treatment planning sealants and the intervention 
period will see a 10% rate. Power was calculated using 50 as the average number of providers per clinic 
in a conservative scenario and 60 in a more realistic scenario using provider data from participating 
clinics. These increases are similar to those seen in similar interventions (Traeger, Lee, Hubscher, & al, 
2019). The analysis assumes different level of intraclass correlation: 0.01, 0.1, and 0.25. The power 
calculation is was done using the “stepped wedge” package in Stata 16 (Hemming & Gerling 2014), 
based on the approach developed by Hussey and Hughes (2007).  
 
 
9.3 
POPULATIONS FOR ANALYSES 
 
The population includes all providers who worked in a KPNW Dental clinic during the non-intervention 
and intervention periods. Providers in the study are dental care providers in KPNW dental clinics.  
 
9.4 
STATISTICAL ANALYSES 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
29 
 
9.4.1 GENERAL APPROACH 
 
Categorical data will be presented as percentages, and continuous data will be presented as means with 
standard deviations. P-values will determine statistical significance with p<0.05 determining statistical 
significance. Shapiro-Wilk tests will be used to determine normality, and corresponding non-parametric 
tests will be used if necessary. A multi-level random effects regression model will be used with providers 
nested within clinics.   
 
Randomization should eliminate the need to adjust for covariates in the analysis of the primary 
endpoint. We will not check for failures of randomization, as this practice has been determined to be 
unsound (de Boer, Waterlander, Kuijper, Steenhuis, & Twisk, 2015).  
 
9.4.2 ANALYSIS OF THE PRIMARY ENDPOINT(S) 
 
The primary end point will be calculated as the change in the providers’ rates of placing or treatment 
planning sealants for occlusal NCCLs from before to after clinic stakeholders are exposed to the 
Deliberative Loop intervention. The PDA guideline includes children, adolescents, and adults. To enable 
comparison with studies following the ADA guideline, in addition to analyzing lesions occurring in 
children, adolescents, and adults, we will also conduct analyses for lesions occurring in just children and 
adolescents. Per the ADA, we will define “children and adolescents” as persons ranging in age from 6 
through 17. The rates, and their difference, will be treated as continuous variables. We will compare the 
change in providers’ rates of placing or treatment planning sealants between the intervention and non-
intervention periods by using a multi-level modeling approach. . We will use a generalized linear model 
or generalized estimating equations to model the intervention effect while nesting sealant outcomes 
within provider and accounting for secular trends, if necessary.  In this scenario, we will treat the change 
in the provider’s rate of placing or treatment planning sealants as the outcome variable and examine the 
clinic-level effects of the intervention.  
 
9.4.3 ANALYSIS OF THE SECONDARY ENDPOINT(S) 
 
Treatment planned sealants resolved. Health Connect data are used to identify occlusal surfaces with an 
NCCL with a treatment plan and plan date and to record subsequent resolution of the treatment plan for 
patients with a dental office visit during the evaluation period. The resolution of the treatment plan may 
include a sealant placement, other procedure, revision of the treatment plan, or no additional 
procedures.      
 
Program costs for the DD intervention. We will quantify total program costs, total costs per clinic, and 
total costs per member per month (PMPM). Total costs to the health system and mean total costs per 
clinic will be assessed, including personnel time, facility space, and materials and supplies used in the 
development, delivery, and clinic implementation of the Deliberative Loop. Resources used beyond a 6-
month study period will be adjusted (discounted) to a single dollar year using an appropriate medical 
services inflation rate. Given that the intervention is expected to change treatment patterns for NCCLs, 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
30 
expenditures for treatment strategies (procedures) on eligible occlusal tooth surfaces will also be 
evaluated during the non-intervention and intervention periods. Intervention and NCCL treatment costs 
during a 6-month clinic evaluation period will be annualized as necessary given the staggered rollout of 
the intervention across participating dental offices.   
 
Cost data for the Deliberative Loop will be obtained for staff time, facility space use, and supplies. We 
use a replication cost approach to estimate the costs of the tested intervention and usual care “as if” the 
health system/clinic was managing the process. We will work with KPNW and PDA staff time to develop 
realistic real-world estimates of the staff time and resources used to develop and print a context setting 
workbook, including staff time that would be needed to conduct an appropriate literature review, 
collect and assess health records data, develop clinic monitoring processes, and prepare, print and 
distribute a completed context setting workbook. This is necessary because study staff has been 
responsible for developing the context setting workbook. Staff time for meetings, context setting 
workbook review, the Deliberative Loop intervention, and relevant clinic follow-up meetings will be 
documented, and average labor costs, by profession type, will be assigned. Facility space for DD session 
meetings and subsequent clinic follow-up meetings will be obtained and valued using local rental costs 
per square foot. Costs for selected implementation interventions (equipment, trainings, CE, staff time) 
will be obtained from clinic managers and other KPNW and PDA staff during the implementation period. 
Treatment costs (procedure costs based on local market paid claims) for eligible tooth surfaces and 
patient-paid costs will be obtained from the electronic health records.  
 
Cost-effectiveness analysis. The incremental CEA will be conducted to quantify the additional 
intervention costs associated with increasing the sealant placement rates during the intervention period 
compared with the non-intervention period. An incremental cost-effectiveness ratio (ICER) will be 
estimated using the following formula: (mTCi – mTCni) / ( mSPRi – mSPRni), where mTCi and mTCni 
represent the mean total intervention costs for intervention and non-intervention periods, including 
NCCL treatment costs, and mSPRi and mSPRni represent the mean cumulative sealant placement rates 
for eligible teeth among patients with a routine care visit during intervention and non-intervention 
periods. We include treatment planned sealants in the numerators. We also estimate ICERs that exclude 
treatment planned NCCLs from the numerator if they remain unsealed at the end of the evaluation 
period. This provides a more conservative estimate of cost-effectiveness.    
 
Treatment costs and CEA results for the health system or clinic may vary based on differences in 
implementation interventions, treatment practices for eligible teeth, and the proportion of treatment 
costs paid by patients. The availability of these data will allow us to conduct exploratory analyses to 
assess the impact of variations in these factors on program costs and CEA results.  
 
We will calculate ICERs and confidence intervals using appropriate statistical methods based on the data 
(Gold, Siegel, Russell, & WEinstein, 1996). We expect differences in both costs and sealant placement 
rates to be unequivocally positive. If, however, the results are equivocal we will conduct simulation 
modeling using bootstrapping and incremental net benefit approaches to assess the effects of model 
input variability on outcomes (Fenwick, O'Brian, & Briggs, 2004; Hunink et al., 2014; Nixon, Wonderling, 
& Grieve, 2010). Incremental net benefit analysis allows adjustment for known statistical issues 
associated with confidence intervals (CIs) for equivocal ICERs by estimating CIs around the organizations 
(payer’s) willingness to pay for the intervention.  
 
The cost and cost-effectiveness analyses do not expect to include downstream sequalae associated with 
the potential arrest of sealed NCCLs or retreatment, due to the limited evaluation period for this study. 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
31 
However, we will examine tooth-level data to determine the frequency of repeat procedures on 
included occlusal tooth surfaces and include these data if appropriate.     
 
 
9.4.4 SAFETY ANALYSES 
 
N/A 
 
9.4.5 BASELINE DESCRIPTIVE STATISTICS 
 
Baseline sealant application rates will be calculated for each clinic as stated above.  
 
9.4.6 PLANNED INTERIM ANALYSES  
 
After each session, we will calculate descriptive statistics for the two survey items assessing fidelity to 
determine the facilitators are maintaining fidelity to the intervention. 
9.4.7 SUB-GROUP ANALYSES 
N/A 
 
9.4.8 TABULATION OF INDIVIDUAL PARTICIPANT DATA 
 
Individual participant data will not be listed by measure and time point. 
 
9.4.9 EXPLORATORY ANALYSES 
 
Target of the Intervention 
 
We will use descriptive statistics to characterize the target of the intervention. 
 
Feasibility and Acceptability 
 
We will use descriptive statistics to quantify the perceptions of the KPNW and PDA management teams’ 
ratings of the acceptability and feasibility of the deliberative loop. 
 
We will use descriptive statistics to summarize the results of the stakeholders’ ratings of acceptability. 
 
Mediator: Voice 
 
We will use mediation analysis to examine the effect of voice as a mediator in the relationship between 
deliberative democracy and guideline adherence. We will use causal inference methods (VanderWeele 
& Vansteelandt, 2014) that have been extended to allow for multiple mediators, as well as exposure-
mediator and mediator-mediator interactions. Their framework includes both regression-based 
approaches and approaches based on inverse probability weighting. We will default to the regression-

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
32 
based approach. If model fitting issues are present, we will use the inverse probability weighting 
approach. 
 
Characterizing the DD Process 
 
To determine whether there are differences in the change in perception of leadership responsiveness as 
a result of the intervention, we will use t-tests. We will use descriptive statistics to summarize the 
results of the measures of barriers identified.  
 
We will conduct an exploratory qualitative analysis to track, document and analyze the progress of 
implementation interventions that clinics select for implementation. These data will enable us to 
understand and contextualize quantitative data collected. If for example there is no clear effect of our 
intervention, qualitative data can help identify differences in the implementation process among clinics. 
 
10 SUPPORTING DOCUMENTATION AND OPERATIONAL CONSIDERATIONS 
 
10.1 REGULATORY, ETHICAL, AND STUDY OVERSIGHT CONSIDERATIONS 
 
10.1.1 INFORMED CONSENT PROCESS 
 
10.1.1.1 CONSENT/ASSENT 
AND 
OTHER 
INFORMATIONAL 
DOCUMENTS 
PROVIDED 
TO 
PARTICIPANTS 
 
We were granted a waiver of signed informed consent for deliberative loop session participants by the 
KPNW IRB. A fact sheet describing in detail the study intervention, study procedures, and risk of harms / 
benefits will outline all elements of consent and will be given to the participant.  
 
10.1.1.2 CONSENT PROCEDURES AND DOCUMENTATION 
 
We were granted a waiver of signed informed consent for the following:  
 
1. Stakeholders who participate in the vanguard study deliberative loop forums. 
 
2. Stakeholders who participate in the stepped wedge deliberative loop forums.  
 
We were granted a waiver of consent for: 
 
1. To enable collection and use of patient data from the electronic health records and 
administrative data for study purposes. The randomized trial uses a data-only process for 
evaluating study outcomes. We do not expect to consent patients to be part of the evaluation. 
Consenting patients is not feasible.    

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
33 
 
Note: Requesting written consent would disrupt clinic operations and deter staff from participating in 
the forum. We will give participants a fact sheet, which provides an overview of the study, study 
procedures, and risks / benefits, and outlines all elements of consent. The fact sheet states: “There is no 
penalty if a study participant decides not to participate, and participation may stop at any time.” We will 
allow time for participants to ask questions. This process will be documented. Any unforeseen need to 
modify the consent process will be reviewed by the KPNW IRB prior to implementation. 
 
10.1.2 STUDY DISCONTINUATION AND CLOSURE 
 
This study may be temporarily suspended or prematurely terminated if there is sufficient reasonable 
cause. Written notification, documenting the reason for study suspension or termination, will be 
provided by the suspending or terminating party to study participants, funding agency, and regulatory 
authorities. If the study is prematurely terminated or suspended, the Principal Investigator (PI) will 
promptly inform study participants, the Institutional Review Board (IRB), and sponsor/funding agency 
and will provide the reason(s) for the termination or suspension. Study participants will be contacted, as 
applicable, and be informed of changes to study visit schedule. 
 
Circumstances that may warrant termination or suspension include, but are not limited to: 
• 
Determination of unexpected, significant, or unacceptable risk to participants 
• 
Demonstration of efficacy that would warrant stopping    
• 
Determination that the primary endpoint has been met 
 
The study may resume once concerns about safety, protocol compliance, and data quality are 
addressed, and satisfy the funding agency, sponsor, IRB, or other relevant regulatory or oversight 
bodies. 
 
10.1.3 CONFIDENTIALITY AND PRIVACY  
 
Participant confidentiality and privacy is strictly held in trust by the participating investigators, their 
staff, the safety and oversight monitor(s), and the sponsor(s) and funding agency. This confidentiality is 
extended to the data being collected as part of this study. Data that could be used to identify a specific 
study participant will be held in strict confidence within the research team. No personally identifiable 
information from the study will be released to any unauthorized third party without prior written 
approval of the sponsor/funding agency. 
 
Certificate of Confidentiality  
To further protect the privacy of study participants, the Secretary, Health and Human Services (HHS), 
has issued a Certificate of Confidentiality (CoC) to all researchers engaged in biomedical, behavioral, 
clinical, or other human subjects research funded wholly or in part by the federal government. 
Recipients of NIH funding for human subjects research are required to protect identifiable research 
information from forced disclosure per the terms of the NIH Policy 
(https://humansubjects.nih.gov/coc/index). As set forth in 45 CFR Part 75.303(a) and NIHGPS Chapter 
8.3, recipients conducting NIH-supported research covered by this Policy are required to establish and 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
34 
maintain effective internal controls (e.g., policies and procedures) that provide reasonable assurance 
that the award is managed in compliance with Federal statutes, regulations, and the terms and 
conditions of award. It is the NIH policy that investigators and others who have access to research 
records will not disclose identifying information except when the participant consents or in certain 
instances when federal, state, or local law or regulation requires disclosure. NIH expects investigators to 
inform research participants of the protections and the limits to protections provided by a Certificate 
issued by this Policy. 
 
10.1.4 FUTURE USE OF STORED SPECIMENS AND DATA  
 
N/A 
 
10.1.5 KEY ROLES AND STUDY GOVERNANCE 
 
Principal Investigator 
KPNW Investigator 
Medical Monitor  
Deborah Polk, PhD  
Assistant Professor 
Jeffrey L. Fellows, PhD 
Senior Investigator 
Kevin McBryde, MD 
Medical Monitor 
University of Pittsburgh  
Kaiser Permanente 
Center for Health Research 
NIDCR 
381 Salk Hall - 3501 Terrace Street 
Pittsburgh, PA  15261 
3800 N. Interstate Ave. 
Portland, OR 97227 
6701 Democracy Blvd 
Bethesda, MD  20817 
412-648-8656 
503-335-6784 
301-594-0170 
Dpolk@pitt.edu 
Jeffrey.Fellows@kpchr.org 
kevin.mcbryde@nih.gov 
 
In addition, a Core Committee provides regular oversight and decision-making for the study. 
 
10.1.6 SAFETY OVERSIGHT 
 
In addition to the PI’s responsibility for oversight, study oversight will be standard oversight. Reporting 
will occur via the annual Research Performance Progress Report and enrollment reports. There will be 
six enrollment reports total. The first will be due two weeks after the Deliberative Loop session is 
completed for the vanguard clinic; and subsequent enrollment reports will be due two weeks after the 
completion of the last Deliberative Loop session in each step of the stepped wedge design. 
 
A data and safety monitoring plan will be implemented by the Principal Investigator to ensure that there 
are no changes in the risk/benefit ratio during the course of the study and that confidentiality of 
research data is maintained. Investigators will meet annually at minimum to discuss the study (e.g. study 
goals and modifications of those goals; progress in data coding and analysis; documentation, 
identification of adverse events or research subject complaints; violations of confidentiality) and address 
any issues or concerns at that time. Any instances of adverse events will be reported to the KPNW IRB 
using standard forms and/or procedures that have been established by the IRB.  
 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
35 
10.1.7 CLINICAL MONITORING 
 
No outside clinical site monitoring will be employed for this study. The Principal Investigator(s) and staff 
will closely monitor the subjects as they progress through the study. They will monitor and evaluate 
study processes and documentation based on the International Council for Harmonisation (ICH), E6: 
Good Clinical Practice guidelines (GCP), and internal quality management plans. The NIDCR reserves the 
right to conduct independent clinical site monitoring as necessary. 
 
10.1.8 QUALITY ASSURANCE AND QUALITY CONTROL 
Quality Assurance (QA) procedures will include the creation of standard operating procedures that 
address the following activities: 
 
• 
Data Review:  
o Recordings of Deliberative forums will be reviewed for fidelity within one week of 
recording by a qualitative research associate.   
o EHR data: CHR implements a quality control process when extracting the data from the 
EHR. 
• 
QA and QC Issues.  The principal investigator will discuss QA and QC issues with relevant study 
personnel and determine appropriate corrective/preventive actions. 
• 
Study Staff Training.  All study personnel will be trained on the study protocol and will 
participate in human research subjects training. Personnel also will be trained on standard 
operating procedures relevant to their roles and responsibilities.  All training will be 
documented. 
• 
For a description of the plans for tracking compliance with the treatment fidelity evaluations, 
please see section 6.2.1. 
 
Each clinical site will perform internal quality management of study conduct, data collection, 
documentation and completion. All sites will follow a common quality management plan.  
 
Should independent monitoring become necessary, the PI will provide direct access to all trial related 
sites, source data/documents, and reports for the purpose of monitoring and auditing by the 
sponsor/funding agency, and inspection by local and regulatory authorities. 
 
10.1.9 DATA HANDLING AND RECORD KEEPING  
 
 
10.1.9.1 DATA COLLECTION AND MANAGEMENT RESPONSIBILITIES  
 
Data collection from individuals: Practitioner / staff data will be collected from recordings of 
the DD sessions and surveys. 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
36 
Data measuring provider adherence to the guideline will be collected from the KPNW data systems. No 
patients will be contacted to collect data. To reduce the risk of breaching confidentiality, records will be 
identified only with a unique ID# on a secure server. We will use the Participant ID number established 
by KP CHR instead of MRNs. The analyst will have sole access to and maintain a link file for patient ID 
and MRN number. Other study team members with access to individual-level data will not have access 
to MRNs.    
 
Data storage plan and access: All individual-level study data obtained from the electronic records 
systems and DD session will be stored on a password-protected file service at Kaiser Permanente Center 
for Health Research (CHR), consistent with established CHR protocols. Access to individual patient data 
will be stored in a study (top) file service that is restricted to CHR analysts, Principal Investigator, and 
project manager. Non-CHR study staff will not have access to the individual-level data stored on the file 
service. At the University of Pittsburgh, all data will be stored on password-protected Box account in the 
cloud. Only the study biostatistician will have access to the raw data. 
 
All study staff will be HSP and FCOI trained and follow CHR protocols for data sharing.  
 
Data sharing: A limited data set will be shared with external (outside of KP) study team members. Data 
use and transfer agreements will be established as necessary before data are shared with external study 
partners. All study data will be sent via secure file transfer (SFT) or encrypted email. No information will 
be released in any publication or presentation that will enable identification of participants. No one 
outside the research process will have access to results about any individual.  
 
Clinical data will be aggregated monthly and de-identified data (sealant placement rates) will be shared 
with DISGO study staff. Periodically during the intervention period and at its completion, a limited data 
set will be compiled and shared with the study statistician for ongoing monitoring and to support 
reporting and manuscript development 
 
10.1.9.2 STUDY RECORDS RETENTION  
 
Once the study is completed and manuscript preparation has ended, all study data will be 
archived and/or destroyed (electronic files deleted; paper files securely disposed) following CHR 
protocols and NIH requirements. 
 
10.1.10 
PROTOCOL DEVIATIONS   
 
This protocol defines a protocol deviation as any noncompliance with the protocol, noncompliance may 
be either on the part of the participant, the investigator, or the study site staff. As a result of deviations, 
corrective actions will be developed by the site and implemented promptly.  
 
It will be the responsibility of the site investigator and project manager to use continuous vigilance to 
identify and report deviations as part of the Medical Monitor Oversight Report. Protocol deviations will 
be sent to the reviewing Institutional Review Board (IRB) per their policies. The site investigator will be 
responsible for knowing and adhering to the reviewing IRB requirements.  

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
37 
 
10.1.11 
PUBLICATION AND DATA SHARING POLICY  
 
This study will be conducted in accordance with the following publication and data sharing policies and 
regulations: 
 
National Institutes of Health (NIH) Public Access Policy, which ensures that the public has access to the 
published results of NIH funded research. It requires scientists to submit final peer-reviewed journal 
manuscripts that arise from NIH funds to the digital archive PubMed Central upon acceptance for 
publication. 
 
This study will comply with the NIH Data Sharing Policy and Policy on the Dissemination of NIH-Funded 
Clinical Trial Information and the Clinical Trials Registration and Results Information Submission rule. As 
such, this trial will be registered at ClinicalTrials.gov, and results information from this trial will be 
submitted to ClinicalTrials.gov. In addition, every attempt will be made to publish results in peer-
reviewed journals.  Data from this study may be requested from other researchers up to five years after 
the completion of the primary endpoint by contacting Jeffrey L. Fellows, Kaiser Permanente Center for 
Health Research.  Considerations for ensuring confidentiality of these shared data are described in 
Section 10.1.3. 
10.1.12 
CONFLICT OF INTEREST POLICY 
 
Any actual conflict of interest of persons who have a role in the design, conduct, analysis, publication, or 
any aspect of this trial will be disclosed and managed. Furthermore, persons who have a perceived 
conflict of interest will be required to have such conflicts managed in a way that is appropriate to their 
participation in the design and conduct of this trial.  
 
10.2 ADDITIONAL CONSIDERATIONS 
 
N/A 
10.3 ABBREVIATIONS AND SPECIAL TERMS 
 
AE 
Adverse Event 
CEA 
Cost-Effectiveness Analysis 
CFR 
Code of Federal Regulations 
CRF 
Case Report Form 
CHR 
Center for Health Research 
CRA 
Caries Risk Assessment 
DCC 
Data Coordinating Center 
DD 
Deliberative Democracy 
DL 
Deliberative Loop 
DSMB 
Data Safety Monitoring Board 
E1/E2 
Enamel Lesions 
EFDA 
Expanded-Function Dental Assistant 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
38 
EHR 
Electronic Health Record 
FCOI 
Financial Conflict of Interest 
HSP 
Human Subjects Protections 
ID 
Identification 
IRB 
Institutional Review Board 
IS 
Implementation Strategies 
ISM 
Independent Safety Monitor 
ITT 
Intention-To-Treat 
KPNW 
Kaiser Permanente Northwest 
MOP 
Manual of Procedures 
MRN 
Medical Record Number 
NCCL 
Noncavitated carious lesion 
NIDCR 
National Institute of Dental and Craniofacial Research 
NIH  
National Institutes of Health 
NIH IC 
NIH Institute or Center 
OHP 
Oregon Health Plan 
PCA 
Program Cost Analysis 
PDA 
Permanente Dental Associates 
PI 
Principal Investigator 
QA 
Quality Assurance 
QC 
Quality Control 
SAE 
Serious Adverse Event 
SAP 
Statistical Analysis Plan 
SOP 
Standard Operating Procedure 
SFT 
Secure File Transfer 
UP 
Unanticipated Problem 
US 
United States 
 
 
 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
39 
10.4 PROTOCOL AMENDMENT HISTORY 
 
Version 
Date 
Description of Change  
Brief Rationale 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
40 
11 REFERENCES  
 
Cavalier, R. J. (2009). Democracy for Beginners. Lebanon, NH: Steerforth Press. 
Clarkson, J. E., Turner, S., Grimshaw, J. M., Ramsay, C. R., Johnston, M., Scott, A., . . . Pitts, N. B. (2008). Changing 
Clinicians’ Behavior: a Randomized Controlled Trial of Fees and Education. Journal of Dental Research, 
87(7), 640-644. doi:10.1177/154405910808700701 
Committee on Revision to the Common Rule for the Protection of Human Subjects in Research in the Behavioral 
and Social Sciences. (2014). Proposed Revisions to the Common Rule for the Protection of Human Subjects 
in the Behavioral and Social Sciences. Washington, DC: National Academies Press. 
de Boer, M. R., Waterlander, W. E., Kuijper, L. D., Steenhuis, I. H., & Twisk, J. W. (2015). Testing for baseline 
differences in randomized controlled trials: an unhealthy research behavior that is hard to eradicate. 
International Journal of Behavioral Nutrition and Physical Activity, 12(4).  
De Vries, R., Stanczyk, A., Ryan, K. A., & Kim, S. Y. H. (2011). A framework for assessing the quality of democratic 
deliberation: Enhancing deliberation as a tool for bioethics. J Empir Res Hum Res Ethics, 6(3), 3-17.  
De Vries, R., Stanczyk, A., Wall, I. F., Uhlmann, R., Damschroder, L. J., & Kim, S. Y. (2010). Assessing the quality of 
democratic deliberation: A case study of public deliberation on the ethics of surrogate consent for 
research. Social Science & Medicine, 70(12), 1896-1903.  
Delli Carpini, M. X., Lomax Cook, F., & Jacobs, L. R. (2004). Public deliberation, discursive participation, and citizen 
engagement: A review of the empirical literature. Annual Review of Political Science, 7, 315-344.  
Drummond, M. F., Sculpher, M. J., Claxton, K., Stoddart, G. L., & Torrance, G. W. (2015). Methods for the Economic 
Evaluation of Health Care Programmes (4th ed.). New York, NY: Oxford University Press. 
Dryzek, J. S., & Niemeyer, S. (2006). Reconciling pluralism and consensus as political ideals. American Journal of 
Political Science, 50(3), 634-649.  
Fenwick, E., O'Brian, B. J., & Briggs, A. (2004). Cost-effectiveness acceptability curves – facts, fallacies, and 
frequently asked questions. Health Economics, 13, 405-415.  
Gold, M. R., Siegel, J. E., Russell, L. B., & WEinstein, M. C. (1996). Cost-Effectiveness in Health and Medicine. New 
York: Oxford University Press. 
Gutmann, A., & Thompson, D. (2004). Why deliberative democracy? Princeton, NJ: Princeton University Press. 
Hunink, M. G. M., Weinstein, M. C., Wittenberg, E., Drummon, M. F., Pliskin, J. S., Wong, J. B., & Glasziou, P. P. 
(2014). Decision making in health and medicine: Integrating evidence and values (2nd ed.). Cambridge: 
Cambridge Unversity Press. 
Kim, S. Y. H., Wall, I. F., Stanczyk, A., & De Vries, R. (2009). Assessing the public's views in research ethics 
controversies: Deliberative democracy and bioethics as natural allies. J Empir Res Hum Res Ethics, 4(4), 3-
16.  
Kuyper, J. W. (2018). The instrumental value of deliberative democracy - or, do wehave good reasons to be 
deliberative democrats? Journal of Public Deliberation, 14(1), Article 1.  
Liang, J., Farh, C. I. C., & Farh, J.-L. (2012). Psychological antecedents of promotive and prohibitive voice: A two-
wave examination. Academy of Management Journal, 55(1), 71-92.  

An implementation strategy for the adoption of an evidence-based guideline for pit-and-fissure sealants Version 
0.7 Protocol 17-077-E 
25 October 2020 
 
Based on the NIH Behavioral and Social Intervention Clinical Trial Protocol Template v2.0 - 20180215 
 
 
41 
Nabatchi, T., Gastil, J., Weiksner, G. M., & Leighninger, M. (2012). Democracy in motion. Evaluating the practice 
and impact of deliberative civic engagement: Oxford University Press. 
Nixon, R. M., Wonderling, D., & Grieve, R. D. (2010). Non-parametric methods for cost-effectiveness analysis: the 
central limit theorem and the bootstrap compared. Health Econ, 19(3), 316-333.  
O'Donnell, J. A., Modesto, A., Oakley, M., Polk, D. E., Valappil, B., & Spallek, H. (2013). Sealants and dental caries: 
Insight into dentists' behaviors regarding implementation of clinical practice recommendations. JADA, 
144(4), e24-e30.  
Polk, D. E., Weyant, R. J., Shah, N. H., Fellows, J. L., Pihlstrom, D. J., & Frantsve-Hawley, J. (2018). Barriers to sealant 
guideline implementation within a multi-site managed care dental practice. BMC Oral Health, 18(1), 17. 
doi:10.1186/s12903-018-0480-z 
Steenbergen, M. R., Bachtiger, A., Sporndli, M., & Steiner, J. (2003). Measuring political deliberation: A discourse 
quality index. Comparative European Politics, 1, 21-48.  
Steiner, J., Bachtiger, A., Sporndli, M., & Steenbergen, M. R. (2004). Deliberative politics in action: cross-national 
study of parliamentary debates. Cambridge, UK: Cambridge University Press. 
Stromer-Galley, J. (2007). Measuring deliberation's content: A coding scheme. Journal of Public Deliberation, 3(1), 
Article 12.  
Tellez, M., Gray, L., Gray, S., Lim, S., & Ismail, A. I. (2011). Sealants and dental caries. Dentists' perspectives on 
evidence-based recommendations. JADA, 142(9), 1033-1040.  
Traeger, A. C., Lee, H., Hubscher, M., & al, e. (2019). Effect of intensive patient education vs placebo patient 
educatoin on outcomes in patients with acute low back pain: A randomized clinical trial. JAMA Neurology, 
76(2), 161-169.  
Van Dyne, L., Ang, S., & Botero, I. C. (2003). Conceptualizing employee silence and employee voice as 
multidimensional constructs. Journal of Managment Studies, 40(6), 1359-1392.  
VanderWeele, T. J., & Vansteelandt, S. (2014). Mediation analysis with multiple mediators. Epidemiolo Methods, 
2(1), 95-115.