CT Dose Optimization and Standardization Endeavor (DOSE) Collaboratory
Phase: NA
Conditions: Ionizing Radiation Exposure, Quality Improvement
Sponsor: University of California, San Francisco

1 
 
August 29, 2017 
 
 
  
 
 
 
CT Dose Collaboratory  
Study Protocol 
A Randomized Trial to Evaluate a Simple Audit and a Multicomponent Intervention to 
Standardize Radiation Doses for CT 
Funded by the National Institutes of Health 
UCSF Committee on Human Subjects Approved Study # 15-17781 
Clinical Trials Registry # NCT03000751 
 
Study PI 
Rebecca Smith-Bindman, MD 
Professor of Radiology, Epidemiology & Biostatistics 
Health Policy, Obstetrics, Gynecology, and Reproductive Medicine 
University of California, San Francisco 
350 Parnassus Avenue Suite 307C 
San Francisco, CA 94122 
Phone: (415) 353-4946 
Fax: (415) 353-2790 
Rebecca.Smith-Bindman@ucsf.edu 
Protocol Date: August 29, 2017 
 
 
 

2 
 
August 29, 2017 
 
 
  
TABLE OF CONTENTS 
 
A. PROTOCOL ABSTRACT 
3 
B. BACKGROUND AND SIGNIFICANCE 
3 
C. STUDY OBJECTIVES 
5 
D. RESEARCH DESIGN AND METHODS 
6 
E. SETTING AND PARTICIPATING INSTITUTIONS 
8 
F. UCSF RADIATION DOSE REGISTRY 
10 
G. INSTITUTIONAL AND FACILITY SURVEYS 
10 
H.  STUDY POPULATION AND UNIT OF ANALYSIS 
15 
I. RANDOMIZATION 
17 
J. STUDY VARIABLES 
18 
K. STUDY ENDPOINTS AND PLANNED ANALYSES 
19 
L. STUDY ARMS 
21 
M. TIMELINE AND TIMELINE OF ASSESSMENTS 
25 
N. STATISTICAL ANALYSIS 
27 
O. SAMPLE SIZE 
29 
P. ETHICAL CONSIDERATIONS AND DSMB 
29 
Q. REFERENCES 
30 
R. APPENDICES 
48 
 
 
 

3 
 
August 29, 2017 
 
 
  
A. Protocol Abstract  
The proposed project is a multi-site collaboration focused on implementing improved standards 
for conducting computed tomography (CT) with respect to the doses used to obtain these 
examinations; studying strategies to implement these standards across diverse healthcare 
facilities using a cluster randomized, stepped-wedge design; and then more broadly 
disseminating the strategies based on what was learned in the implementation trial; and 
conducting an observational study of dissemination during years 4-5. The broad goal of this 
work is to improve the safety of CT by lowering the doses to which patients are exposed and 
thus reduce the number of cancers that result from these exposures. This project is focused on 
determining the best approaches to accomplish this. We will study the change process itself as it 
varies across different types of healthcare facilities with varying organizational structures to 
understand the best strategies to adapt our interventions based on context and other factors to 
improve its chances of success. This project will be conducted in partnership with diverse 
healthcare delivery institutions in the U.S., Asia, and Europe. All participating institutions have: 
Radimetrics installed, a Bayer Health software product for radiation dose monitoring; identified 
a site leader who is an institutional champion for change; and demonstrated an organizational 
commitment to dose optimization.  
B. Background and Significance 
Despite how frequently CT is used (75 million CT examinations are conducted annually in the 
U.S.) and the high doses of radiation it delivers (up to 500 times higher than a chest x-ray), there 
are few standards for the conduct of CT examinations. This has resulted in excessive variation 
in the doses that patients receive when they undergo CT, and the routine use of doses higher 
than needed for medical diagnoses – doses associated with an increased risk of cancer. Several 
recent studies have confirmed an elevated risk of cancer associated with exposure to CT.1,2 
Although there has been increasing attention paid to CT use from consumer advocates, medical 
groups, quality organizations, and state and national legislators around improving the safety of 
medical imaging, little progress has been made: our recent research documents that radiation 

4 
 
August 29, 2017 
 
 
  
doses remain higher than needed for diagnosis.3,4 The manufacturers are developing lower dose 
devices, but these will not replace existing machines for decades or longer as over 10,000 
machines are currently in operation. Thus, current machines need to be made safer.  
No U.S. professional or governmental organization is responsible for collecting, monitoring, or 
reporting patient dose information for CT, and no comprehensive standards or guidelines cover 
conduct of CT studies. The general principle is that doses should be "as low as reasonably 
achievable,"43 but no guidelines define reasonable or achievable. In the absence of explicit 
guidelines, practice variation potentially introduces unnecessary harm from excessive radiation 
and a practice climate in which almost any dose is acceptable. The FDA issued a national 
advisory for hospitals to carefully check their CT protocols40 after patients received brain 
radiation doses approximately 1000-times higher than the average dose for brain CT. One 
hospital reported “intentionally" using high doses to get clearer images,41 highlighting the lack 
of standards. The absence of guidelines on radiation dosing in the U.S. contrasts with quality 
assurance programs in Europe that have been in place for more than a decade.44 European 
programs use diagnostic reference levels45-48 that help facilities identify when their average 
doses are too high. These programs define upper limits for radiation dose that should be 
exceeded only for justified reasons documented in medical records. The UK has had CT 
mandates for more than 15 years, with extensive radiation dose monitoring at institutional, 
local, and national levels. However, concrete benchmarks have been created even in the UK for 
only a few routine imaging protocols and have not been updated since 2003, generating interest 
in standardizing and optimizing CT radiation doses across a range of indications and protocols. 
This study will draw on the UK experience and collect data on a large number of CT indications 
with a goal of creating benchmarks covering a larger proportion of protocols and practices. 
Diverse organizations agree that patients must receive the lowest radiation dose possible to 
achieve the necessary medical benefit from CT.42 Organizations such as the American College of 
Radiology,49 the Institute of Medicine (IOM),20,50 the Center for Medicare and Medicaid 
Services,51 39,40 the Joint Commission on Health Care Accreditation,52 U.S. Congress,53,54 and the 
Image Gently social marketing campaign for children55 all stress safer CT practices. Recognizing 

5 
 
August 29, 2017 
 
 
  
the potential risks of dose variation and radiation overdosing, the FDA plans to increase 
oversight of CT39,40 and called for institutions and professional societies to create diagnostic 
reference levels that, if exceeded, would trigger investigative action and dose-reduction 
strategies.39 Achieving safer CT also requires more quality information on current radiation 
exposures—data that we collect in a related, PCORI sponsored project. The next step is 
generating, applying, and evaluating validated dose-lowering and optimizing strategies across 
diverse healthcare institutions—the goal of this proposal.  
Concrete data are needed on successful strategies to lower and optimize CT radiation doses. 
This will require balancing the importance of CT to clinical care with the risk of radiation injury 
from medical errors and the potential of elevated cancer risk even when protocols are followed. 
The first step is lowering CT doses when feasible. Evidence suggests average CT doses could be 
reduced by 50% or more without reducing diagnostic accuracy.56 In this project, we will 
leverage the expertise of nationally respected scientists and radiation physicists to assess and 
reduce CT doses at facilities and standardize and optimize protocols across participating 
organizations. We will compare different strategies in a clustered randomized controlled trial 
(RCT) of implementation using a stepped-wedge design and work closely with our partner 
organizations to study and implement lasting and sustainable changes to improve CT safety. 
C. Study Objectives 
C.1 Primary Objective:  
• 
Using a stepped-wedge cluster randomized controlled trial (RCT), evaluate the effects of 
a simple audit and of a tailored multicomponent intervention on lowering facility-level 
radiation doses from diagnostic CT imaging, relative to baseline, for head, chest, and 
abdomen/pelvis CT.  We will randomize facilities based on time zone and independence 
of facilities within their associated healthcare organizations.   
o Primary outcome: Effective dose 
 

6 
 
August 29, 2017 
 
 
  
C.2 Secondary Objectives: 
(1) Evaluate the effects of a simple audit and a multicomponent intervention on lowering 
facility-level radiation doses using additional dose metrics, including CTDIvol 
(Computed Tomography Volumetric Dose Index) and DLP (Dose Length Product). 
o Outcomes: CTDIvol, DLP 
(2) Evaluate the delayed and/or longer-term effects of the two interventions on effective 
dose, CTDIvol and DLP. 
(3) Identify facilitators and barriers (assessed through surveys of participating facilities) 
associated with successful and failed implementation of dose optimization (i.e., 
improvements in doses after each intervention). This will be assessed following 
completion of the simple audit, multicomponent intervention, and one year following 
the multicomponent intervention.  
o Outcomes: Effective dose  
D. Research Design and Methods  
D.1 Study Design  
We will conduct a pragmatic stratified randomized controlled trial (RCT) of two interventions 
using a two-phase stepped-wedge design. We will first evaluate the effect of a single-
component intervention (simple audit). Second, we will evaluate the effect of a multicomponent 
intervention (audit + tailored recommendations + quality improvement collaborative calls 
combined with Change/Implementation Team support) with considerable external support and 
a close relationship with the University of California leadership team, but local flexibility.  We 
will also assess whether participation in an in-person meeting (where education in quality 
improvement is emphasized) enhances the impact of the audit and/or the multicomponent 
intervention. We will determine if the simple, inexpensive audit and feedback approach is as 
effective as the intensive, multicomponent intervention at reducing facility-level mean radiation 
dose or the percentage of examinations with doses above the benchmark compared to baseline. 
Since audit with feedback is expected to be only weakly effective, we will implement it first. The 

7 
 
August 29, 2017 
 
 
  
second intervention will give tailored feedback on needed changes plus guidance using quality 
improvement methods that facilitate organizational change.   
A total of 116 facilities from one of 20 parent institutions will participate in the trial. The 
individual facilities will be the unit of comparison in the trial. However, in order to ensure that 
facilities that are highly integrated with each other (for example, share staff and management) 
do not get randomized to separate groups (which would be impractical to implement and lead 
to complete cross over) we assessed the relative independence of each facility so that we can 
keep interdependent facilities in the same randomized units (which in some cases will include 
more than one facility). Thus, the unit of comparison is the individual facility, whereas the unit 
of randomization is the facility cluster (see G below). This resulted in 84 units (facility clusters) 
to be randomized.  
The 84 facility clusters will be randomly divided into three tracks, which will undergo the audit 
and multicomponent intervention in a staggered fashion. In addition to this, one of the three 
randomization tracks will be invited to participate in the in-person meeting (where focused 
education on quality improvement and strategies for dose optimization will be provided) to 
coincide with provision of the simple audit, and one randomization track will be invited to the 
in-person meeting to coincide with the multicomponent intervention. By offering participation 
in the in-person meeting at different times in relation to the audit and multicomponent 
intervention, we will assess whether the in-person meeting enhances the impact of each 
intervention on outcomes. 
For our secondary outcomes focused on assessing factors associated with optimization in dose, 
a mixed-methods approach with interviews and surveys will determine factors and strategies 
associated with successful implementation. Surveys will combine material from existing 
validated surveys (such as the Change Process Capability Questionnaire (CPCQ)) adapted to 
radiology-specific quality improvement to measure organizational priority and readiness for 
change, and from the Physician Practice Connections–Readiness Survey (PPC-RS) to measure 
implementation changes over time.  

8 
 
August 29, 2017 
 
 
  
E. Setting and Participating Institutions 
The study will be conducted with healthcare institutions that already utilize Radimetrics 
software, and that contribute data to the UCSF Radiation Dose Registry (UCSF-RDR, see F 
below). Purchase of Radimetrics demonstrates our collaborators’ commitment to assessing and 
optimizing CT doses, and facilitates the standardized collection of radiation dose data across 
diverse institutions. Any healthcare institution that uses Radimetrics, and is able to 
electronically transfer data to the UCSF-RDR using this software (some institutional firewalls 
preclude doing this), is eligible for inclusion in the study. All Radimetrics customers were 
invited to participate in the project through an email invitation that was sent broadly to each 
customer. All customers interested in participating and who could complete the logistical 
requirements (establish data connections, complete data use agreements, complete institutional 
review board approvals, create a local quality improvement team and agree to complete the 
study aims) were invited to participate in the trial.  
The healthcare institutions included below are participating in the study. Some of the 
institutions listed below include a single physical facility, whereas others have many associated 
facilities. While we encouraged all facilities associated within each healthcare institution that 
contribute a minimum number of examinations (120 scans within a three month period) to 
participate, a healthcare institution could participate even if only some of its facilities wanted or 
were qualified to participate. The included institutions provide geographic, socioeconomic, age, 
and racial/ethnic diversity of patients; contain representation from the major manufacturers and 
types of equipment used and diversity in delivery models. Healthcare institutions outside the 
U.S. are included, as CT efforts around dose optimization have been going on for far longer in 
Europe, and doses may be different from those in the U.S. The institutions noted with an 
asterisk are located outside the U.S.  
 
 

9 
 
August 29, 2017 
 
 
  
 
Institution 
 
Abbreviation 
Number of 
Facilities 
(*) 
University Hospital of Basel 
 
BAS 
2 
 
Center for Diagnostic Imaging 
 
CDI 
41 
 
Community Health Network 
 
CHN 
8 
 
Children’s Mercy Hospitals and Clinics 
 
CMH 
2 
 
City of Hope 
 
COH 
1 
 
Einstein Medical Center 
 
EMC 
4 
(*) 
University of Duisburg-Essen 
 
ESN 
4 
 
East Texas Medical Center 
 
ETMC 
9 
 
Emory Health System 
 
EU 
11 
 
Henry Ford Health System 
 
HFH 
9 
 
Huntsville Hospital 
 
HH 
5 
 
Miami Children’s Hospital 
 
MCH 
1 
 
Mount Sinai Health System 
 
MSH 
1 
(*) 
Maastricht University Medical Center 
 
MUMC 
1 
 
Olive View – UCLA Medical Center 
 
OVMC 
1 
(*) 
John Radcliffe Hospital, National Health Services 
OXF 
3 
 
San Francisco Veteran’s Administration Health 
Care System 
 
SFVA 
1 
 
St. Joseph Health System 
 
SJHS 
7 
(*) 
St. Luke’s International Hospital 
 
SLIH 
3 
 
University of Virginia Health System 
 
UVA 
2 
 

10 
 
August 29, 2017 
 
 
  
F. UCSF Radiation Dose Registry 
A secure central CT radiation dose registry, the UCSF Radiation Dose Registry (UCSF-RDR), 
has been created at UCSF to pool data across the collaborating institutions and facilities. This 
registry was funded by a separate research contract from the Patient Centered Outcomes 
Research Institute (PCORI). Each institution or facility pools data from consecutive CT scans 
performed at their institution or facility on their local server, and these data are transferred and 
pooled at UCSF. All facilities that contribute to the registry currently use Radimetrics - a web-
based, medical imaging, radiation dose-monitoring software product owned by Bayer that 
quantifies the radiation delivered in CT scans. Radimetrics links to medical image archiving and 
communication systems to collect examination information and CT dose data delivered. Data 
come from the output of individual scanners using DICOM (Digital Imaging and 
Communications in Medicine) headers and structured reports on scanners when available, or 
optical character recognition if necessary. Data are collected for each radiating event (series). 
Monte Carlo simulation estimates absorbed and effective doses. The software also collects the 
image for calculating size-specific doses. These details allow valid, nuanced, and readily 
understandable metrics to be assembled for radiation dose within each facility.  
The radiation data stored within the CT examination file (stored in DICOM data) are pooled 
directly from the machines where the CT scans are performed, or from the PACS (Picture 
Archiving and Communication Systems) where the scans are reviewed and stored on the local 
Radimetrics instance. The data are stripped of identifiers and in real time are then submitted to 
the UCSF-RDR. All identifying information, other than study date and time, is stripped or 
obfuscated, prior to submission to the registry. The data now stored in the registry allow us to 
access facility–specific radiation dose data on all consecutive CT examinations conducted at 
each of the collaborating facilities. 
G. Institutional and Facility Surveys 
Several surveys will be administered to each facility to understand the 
organizational factors that may influence their dependence on other facilities within 

11 
 
August 29, 2017 
 
 
  
their overall umbrella healthcare institution and thus how they will be randomized 
in the trial, and to assess institutional factors and facility factors that might influence 
dose indices and implementation of dose optimization following audits.  
We will administer two types of surveys to capture facility-level characteristics: 
   1) structural and organizational aspects of each facility's CT imaging work flow 
(organizational survey) 
   2) cultural and behavioral aspects of each facility's approach to CT imaging, especially as 
regards patient safety, quality improvement, power and decision-making, etc. (implementation 
survey) 
We administered the organizational survey prior to randomizing facilities.  This survey 
collected information on structural and organizational patterns at our various facilities, 
including their size and type, numbers of personnel of different types (e.g. CT technologists, 
physicists, radiologists, residents, administrators, etc.), number of CT scanners, number of CT 
imaging protocols, how often protocols are reviewed, and who has the authority to change 
them.  The organizational survey includes a combination of quantitative and qualitative 
questions.  
We will administer our implementation (or "cultural") survey four times over the course of the 
study.  This survey will collect information on imaging and operational practice patterns that 
could affect dose levels and abilities or willingness to change doses.  The implementation 
survey includes a series of likert-scale questions that ask respondents for their opinions about 
practice patterns at their facility. Collectively we anticipate that trends across a number of 
conceptual "domains" could have correlational and predictive power related to CT dose. 
We will use the surveys to understand and measure facility-level characteristics as they pertain 
to CT imaging. The second primary purpose of the surveys is to assess how facility-level 
characteristics correlate with CT dose levels at baseline.  For example, are their systematic 
relationships between the number of CT imaging protocols at a facility and their CT dose levels. 

12 
 
August 29, 2017 
 
 
  
The third primary purpose of the surveys is to assess how facility-level characteristics predict 
changes in CT dose levels throughout our study.  For example, we might hypothesize that 
facilities with a strong "culture" of continuous quality improvement or CT protocol 
reassessment could experience greater reductions in CT dose in response to the interventional 
components of our study. 
G.1 Organizational Survey 
We are including 116 individual facilities in the project. While several individual facilities might 
be owned or run by a single umbrella organization (healthcare institution), they all could 
function similarly (collectively) or differently (independently), or two could function jointly and 
a third separately.  
The primary unit of comparison will be the individual facility. The primary unit for 
randomization will be the facility cluster, referring to a group of facilities within a healthcare 
institution that have a high degree of integration so that separate randomization would not be 
possible without contamination. We identified 84 facility clusters. 
Determining facility clustering: 
Highly integrated facilities within a single institution (e.g. with technologists, radiologists, and 
physicists moving freely between facilities/hospitals and sharing process for CT) will be 
considered a single facility “cluster” for the purpose of determining our randomization units. 
On the other hand, independently functioning facilities within a single system will be treated as 
separate facility clusters for the purpose of randomization. 
We determined the independence of the facilities using an organizational survey and a derived 
CT facility independence index. The organizational survey highlights important features at each 
healthcare institution that we anticipate could influence how radiation doses relate to CT 
practices. We surveyed parent institutions and their component facilities, asking questions 
designed to gauge how much they functioned independently from their parent system, and 

13 
 
August 29, 2017 
 
 
  
whether they should be considered clustered with other facilities for the purpose 
randomization.  
Organizational survey questions that we used to derive the CT facility independence index 
focused on the processes used to develop CT protocols, sharing radiology personnel, sharing 
best practices, and requiring external approval on protocols. Using a 7-point scale (scored from 
0 to 6 points), facilities with a higher score were deemed to be more independent than facilities 
with a lower score. Facilities that scored 3 points or higher were considered independent (i.e. 
little sharing of process or staff, with significant internal decision making on protocols). When 
facilities were considered "dependent" (scoring lower than 3 points), they were grouped with 
the other facilities in their parent institution with which they identified the sharing of staff and 
practices, forming a facility cluster. 
Facilities will be randomized in the trial by facility cluster (the group of facilities considered 
dependent on another). However, the primary unit of comparison to assess the impact of the 
interventions will be the individual facility. When reports are provided to each facility, they will 
be shown the doses at their own facility, their own doses in comparison to all other facilities 
that are part of their umbrella institution, and in comparison to other all other facilities in the 
trial.  
G.2 Institutional and Facility Implementation Surveys and Interviews 
Since we are implementing the strategies in highly diverse environments, we will study 
institutional and facility factors associated with doses at baseline and change in dose after the 
different interventions. The goal is to understand the hospital environments and cultures that 
are conducive to adopting strategies to improve doses. We will assess the relationship between 
hospital characteristics (such as readiness for change, and hierarchical arrangement of how CT 
protocol decisions are made) and the likelihood of the interventions succeeding to lower and 
optimize doses.  

14 
 
August 29, 2017 
 
 
  
Surveys and interviews will be conducted at each facility because decision-making, 
organizational culture, and details of how CT protocols are set can differ between facilities even 
if they share staff and processes. For example, we will survey facility and radiology leaders to 
understand personal priority for optimizing CT radiation doses, perceived value of audit 
reports, perceived barriers to improvement, and relevant quality improvement strategies in 
development or implementation. We will ask the radiology administrative leader about training 
of personnel performing CT scans and proportion done outside normal workdays. We will ask 
about their reasons for purchasing Radimetrics, its use thus far, whether it has met expectations, 
and its impact on CT practice. We also will ask questions about the culture of quality 
improvement. The survey will include questions from the CPCQ, PPC-RS. 
When assessing the relationship between survey results and success/failure of implementation, 
each facility will be considered separately.   
We will ask collaborators to engage organizational leaders at their facilities, including at a 
minimum a non-radiology health system leader, chief radiologist, lead CT technologist, 
radiation safety expert, and departmental administrator. Leaders will participate in surveys and 
ultimately help implement systems-based strategies (such as locking CT protocols).  
We will use an adapted PPC–RS from the National Committee for Quality Assurance that 
focuses on practice system infrastructure. This will include assessment of factors involved in 
dose management (e.g., systematic review of individual case dosages, department policies to 
standardize dosage, radiologist and technologist agreements to follow policies, processes to 
address staff who do not achieve dosage improvements, staff and patient education about 
radiation risks, etc.). The follow up surveys will try to include, at minimum, a radiologist and 
technologist from each facility.  
G.3. Administration of the Surveys 
Upon agreement to participate in the study, each healthcare facility was sent a link to 
complete an organization survey, and a request to provide a graphical depiction of 

15 
 
August 29, 2017 
 
 
  
their organizational structure. Each facility within the institution was asked to 
complete a questionnaire to provide further context about their relationship with the 
larger parent institution and other facilities within their institution. The 
questionnaire focus on staff composition, protocol development and maintenance, 
making changes to protocols, level of practice sharing, and commitment to dose 
optimization. Upon completion, questionnaires were returned to study coordinators, 
who reviewed and followed up with facilities whenever further information or 
clarification was necessary. 
Once the study begins, we will email invitations to complete a REDCap (Research Electronic 
Data Capture) survey in year 1, prior to the beginning of the RCT. The survey invitation will be 
sent to an authority in the radiology department: the site lead champion, radiologists and 
technologists at baseline.  REDCap surveys will again be sent to each hospital periodically (after 
the simple audit and prior to the multicomponent intervention, and 4-6 months and then 12 
months after the multicomponent intervention) to inquire whether the recommended CT 
improvements were implemented and/or remain in place. The radiologists and technologists 
will be invited to participate in our REDCap survey.  We will offer incentives, in the form of 
raffle drawings for prizes, to those who complete the survey within the given time periods. 
H.  Study Population and Unit of Analysis 
CT doses will be measured at the examination (i.e., encounter) level.  Because examinations 
from the same machine, facility, and institution are structurally nested (hence non-
independent), we will use appropriate statistical techniques to account for this correlational 
structure. 
H.1 Included Patients and CT Examinations 
We will include consecutive patients who undergo a diagnostic CT at a participating facility 
from November 2015 to December 2018. We will include CT scans of the head, chest, and 

16 
 
August 29, 2017 
 
 
  
abdomen and pelvis. As an exploratory analysis, we will also evaluate combined 
chest/abdomen/pelvis (CAP) exams.  
H.2 Excluded Patients and CT Examinations 
• 
Patients over the age of 99 years will be excluded. 
• 
Musculoskeletal (i.e., spine and extremity) CT scans will be excluded because the 
methodology for assessing their doses is less developed, and organ doses and cancer 
risk are lower for these examinations.  
• 
CT scans of combined areas (other than combined chest/abdomen/pelvis) will be 
excluded because of the complexity of estimating dose.  
• 
CT scans conducted at external institutions and sent to the Picture Archiving and 
Communication System (PACS) systems of the participating facilities will be excluded 
because these do not reflect CT practices at participating facilities.  
• 
CT scans performed to guide radiation therapy will be excluded 
• 
CT combined with Positron Emission Tomography  (PET-CT) scans will be excluded.  
• 
CT scans used to guide procedures and biopsies will be excluded. 
H.3 CT Examination / CT Encounter as the primary unit of dose assessment 
Data will be assembled from the UCSF-RDR for patient level variables and CT examination 
(more clearly conceptualized as an encounter) level variables. An encounter is a complete CT 
study, which can include imaging of more than one anatomic area and can also include several 
irradiating events (such as a scan with intravenous contrast and a scan without intravenous 
contrast).  The unit of CT dose assessment will be the examination (encounter) and include all 
irradiating events that occurred as part of the examination. If a patient undergoes two or more 
scans as part of that examination (encounter) – for example a scan with contrast and a scan 
without contrast – the dose metrics will be combined and summed for that examination 
(encounter). 
 

17 
 
August 29, 2017 
 
 
  
H.4 Survey and Interview Participants 
We will ask collaborators from each institution to identify individuals from each facility to 
participate in the surveys and interviews. We will ask the collaborators to engage 
organizational leaders at their facilities, including where possible a non-radiology health system 
leader, chief radiologist, radiologist, lead CT technologist, technologist, radiation safety expert, 
medical physicist and departmental administrator. Some of the participating facilities are very 
small (i.e. have only a single employee such as a radiology technologist), whereas others are 
large, and thus the number of available individuals varies. The individuals who participate in 
surveys will also ultimately help implement successful strategies (such as locking CT protocols).  
I. Randomization 
The unit of randomization will be the facility cluster. Facilities will be randomly assigned to one 
of three tracks that determine the timing of when they receive the audits, in-person meeting, 
and multicomponent intervention. Randomization will be conducted using matching 169,170 and 
re-randomization 171 to ensure balance based on the following facility-level factors:  
1) Total number of scans between December 1, 2015 and March 20, 2016. (measure of volume) 
2) Total number of pediatric scans in the same time period. 
3) Mean DLP of Abdomen, Chest, and Head scans in the same time period (to account for dose) 
4) Proportion of Abdomen, Chest, and Head scans over benchmark (75th percentile of DLP) in 
the same time period. 
5) Proportion of Abdominal Diameters that fall in the 1st-5th (quintile) size categories of overall 
population (to account for patient size) 
6) Geographic location (Asia, Europe, West US, South US, Midwest US, Northeast US) 
7) Time zone 
8) A number of other facility characteristics based on self report including whether facility is: 
    - A children's hospital  
    - In the U.S.  
    - Academic  
    - A trauma center  
    - Public / Private  
-  Community hospital  
    - Acute care facility  
    - Tertiary referral hospital 
    - Outpatient imaging facility  

18 
 
August 29, 2017 
 
 
  
 
We will also apply these two additional conditions:  
 
1) All facilities of the same "facility cluster" must be in the same track.  
 
2) All facilities in a given track must be able to participate in phone calls at reasonable work 
hours, defined as 7 am – 5 pm local time. This means that some geographic locations cannot be 
in the same track. 
A biostatistician not affiliated with UCSF, nor with other collaborators on this project, will 
perform the randomization.  
J. Study Variables 
Data will be assembled via the UCSF-RDR for patient-level variables and CT examination-level 
variables and through surveys of each participating facility to assess facility - and institutional-
level variables.                 
J.1 Patient-level Variables 
The patient-level variables of age and sex and anatomic area imaged will be extracted from the 
Radiology Information System (RIS) and PACS through the Radimetrics software and are 
available in the UCSF-RDR.  A mid-scan diameter extracted from the CT image will be used as a 
surrogate for patient size and weight. 
J.2 Examination/encounter-level Variables 
The specific protocol used for every imaging examination is recorded within the RIS and PACS 
and will be imported to the central server by Radimetrics. Anatomic area definitions will 
include head, neck, chest, abdomen/pelvis, combined chest/abdomen/pelvis (CAP), spine 
(cervical, thoracic, and lumbar), and extremity (although only some will be used in this study as 
described above). We will also develop and validate approaches for defining these anatomic 
areas using other collected variables, such as study description and protocol name.  Setting 
(inpatient, outpatient, emergency department [ED]), time/day when examination was 

19 
 
August 29, 2017 
 
 
  
conducted (time of day, day of week, and month), equipment (machine manufacturer, model 
and year), will be extracted from Radimetrics and used to assess as potential confounders.  
J.3 CT Dose Metrics 
The radiation dose data will be assembled and stored for each radiation event. Radimetrics 
extracts direct measures of dose for each radiating event from the CT reports (CTDIvol and 
DLP) and calculates effective dose. Radiation dose indices will include effective dose, CTDIvol, 
and DLP. Our primary outcome measure will be effective dose. CTDIvol and DLP will be 
analyzed as secondary outcomes.  
Assessing diagnostic accuracy is outside the scope of this project. However, we will ask 
radiologists on surveys about their satisfaction with image quality as part of the pre- and post-
intervention interviews and surveys and will assess the association between satisfaction and 
dose.   
J.4 Calculated Variables 
Dose indices will be assessed and presented graphically and in tabular format. They will be 
calculated prior to and following the simple audit report and the multicomponent intervention. 
The dose indices will be stratified by age (child, adult), anatomic area (head, chest, 
abdomen/pelvis, combined chest/abdomen/pelvis), and machine manufacturer and model, as 
well as geographically, e.g. by U.S. and non-U.S. location. Diagnostic reference levels (DRLs) 
will be created overall at baseline across all facilities using the interquartile range (IQR) and 75th 
percentile. The proportion of examinations above benchmarks will be calculated against fixed 
thresholds created during a single baseline period. 
K. Study Endpoints and Planned Analyses 
Endpoints: 
The primary outcome is effective dose.  

20 
 
August 29, 2017 
 
 
  
The Secondary outcomes include two other dose metrics, CTDIvol and DLP.  
During each time period and within each anatomic area, we will assess the mean dose and the 
proportion of examinations above the baseline 75th percentile benchmarks (see above), for each 
facility. We will only assess doses in children if a facility has at least 12 examinations for at least 
one machine within at least one anatomic area during the audit period. 
Planned analyses: 
Primary Analysis 
1. Effective dose. 
   Time frame (see Section M for details): 
    
Baseline 1: 
9-12 months prior to simple audit; 
    
Follow-up 1: 3-6 months following 4 weeks washout after simple audit; 
 
Baseline 2: 
2-6 months prior to multicomponent intervention; 
Follow-up 2: 3-6 months following 2 weeks washout after multicomponent 
intervention; 
 
Within each anatomic area, the mean effective dose and the proportion of CT scans with an 
effective dose over the benchmark during the Follow-up period will be compared to doses 
during the Baseline period.  The benchmark for each anatomic area is defined as the 75th 
percentile of the dose distribution during Baseline 1.   
 
Secondary Analysis  
 
1. Volumetric Computed Tomography Dose Index (CTDIvol). 
   Time frame and comparisons same as for Effective dose described above 
 
2. Dose Length Product (DLP). 
   Time frame and comparisons same as for Effective dose described above 

21 
 
August 29, 2017 
 
 
  
3. Delayed effects: Effective dose, CTDIvol, DLP 
   Time frame (see Section M for details): 
    
Baseline 1: 
9-12 months prior to simple audit; 
    
Follow-up 1: 3-6 months following 4 weeks washout after simple audit; 
 
Baseline 2: 
2-6 months prior to multicomponent intervention; 
Follow-up 2 : 3-6 months following 2 weeks washout after multicomponent 
intervention; 
 
Follow-up 3 : 12-15 months after multicomponent intervention  
 
Within each anatomic area, the mean effective dose and the proportion of CT scans with an 
effective dose over the benchmark during Follow-up period 3 will be compared to doses during 
the Baseline periods prior to each intervention and the Follow up periods after each 
intervention.   
 
4. Facilitators and Barriers associated with Effective dose and successful and failed 
implementation of dose optimization 
 
We will use a linear mixed model analyses of changes in dose, and we will use mixed-methods 
approaches to identify facility-level factors (assessed through surveys of participating facilities) 
associated with effective dose levels and facilitators and barriers associated with successful and 
failed implementation of dose optimization (i.e., improvements of doses during Follow-up 
periods 1, 2, and 3 relative to Baseline periods 1 and 2).  
 
L. Study Arms (Tracks) 
The study will divide facility clusters into three separate tracks. Each facility will first receive 
the simple audit, followed later by the multicomponent intervention, and will be staggered as to 
when they begin each intervention. Two of the three randomized tracks will be invited to 
participate in an in-person meeting, which will be timed to coincide with the audit for one of 

22 
 
August 29, 2017 
 
 
  
the tracks and to coincide with the multicomponent intervention for one of the tracks. The third 
track will serve as a control (i.e., no in-person meeting). This will permit assessing whether the 
in-person meeting enhances the impact of the audit and/or multicomponent intervention. 
L.1 Audit 
Audits will be sent to each facility summarizing their own doses and comparing them to the 
other facilities in their randomized track. The audits will be stratified by anatomic area and age 
group (adult/child) using each of the identified dose metrics. Facilities will be provided dose 
metrics in comparison to other facilities in their facility cluster, the healthcare institution, and 
overall compared with other facilities in their track and in the entire study. The audits will 
summarize data from a time period prior to provision of the audits and multicomponent 
interventions.  
Each audit report will compare the facility against other facilities within their institution and to 
the other facilities/institutions in the study. The comparison will be made based on mean as 
well as median radiation doses, and the proportion of examinations above the baseline 75th 
percentile of radiation dose. Results will be provided both unadjusted and adjusted for patient 
size (using mid scan diameter as a surrogate for patient size). The audit reports will include 
descriptive tables and figures showing protocol-specific measures of radiation dose, outcome 
metrics indicating how their radiation doses rank against other facilities, stratification of results 
by age and anatomic area and results by the machine make and model. The audits will also 
provide information on patients who received very high doses.  
After facilities receive the initial audit report, there will be a four-week (“washout”) period 
where facilities will have time to make changes to their imaging processes.  For example, 
facilities can hold staff meetings or make any changes that could lead to dose optimization prior 
to when we assess their doses the next time.  Facility leads will be responsible for developing 
plans for using audit data for quality improvement with our targeted suggestions; for example, 
meeting with technologists to review protocols or modifying existing protocols for a particular 

23 
 
August 29, 2017 
 
 
  
anatomical area. This phase will test whether complete, comparative data alone can effect 
changes in dose levels.  
Audit reports will be sent to “facility lead champions” (as requested/noted by the site PI) and 
facilities will be encouraged to share the information broadly with other contributors to 
radiologic practices at their facility or institution.  
L.2. Multicomponent Intervention 
The multicomponent intervention will, in addition to the audit reports, provide a more detailed, 
tailored, facility-specific feedback (more detailed audit report) that will match the information 
provided on the audits to workable interventions and actionable suggestions for lowering 
doses. For example, not only will high dose protocols be identified, but also facilities will be 
provided with guidance on technical parameters that they could alter to match the performance 
of the best performing facilities. Further, the multicomponent intervention will include an 
invitation to participate in quality improvement collaborative calls where facilities will meet via 
teleconference regularly (weekly over 8 weeks) to learn about quality improvement and share 
experiences and challenges with adopting the various practices that we are promoting. These 
regular meetings between a UCSF established Change Team and a hospital created 
Implementation Team will help guide the implementation of the key aspects of the intervention.  
Facilities will receive recommendations targeted to performance, which will be sent to 
individuals, identified by the PI at each facility, but will ideally include facility leads, heads of 
radiology departments, and Implementation Team leaders. Recommendations will focus on 
specific, alterable, technical factors (e.g., scan length); specific protocols to use more or less 
frequently (e.g. multiphase protocols); protocol modifications based on cross-facility analyses, 
etc. Recommendations will be tailored to the facilities practices. The audit reports will be a 
starting point for discussions about strategies and techniques to improve CT dose levels. 
Change/Implementation Team meetings and Quality Improvement Collaboratives: 
The University of California based Change Team includes physicists, lead technologists, 

24 
 
August 29, 2017 
 
 
  
radiologists, an administrator with extensive experience in departmental change who will be 
the liaison to facilities, and a leader in implementation research who knows how to implement 
change. Our facility lead champions will assemble local Implementation Teams based on the 
facility structure (e.g., who makes decisions about protocols or monitors legal issues). Members 
of the Change Team will advise facility leads on the creation of the Implementation Team. 
Implementation Teams will use baseline audit results to convey the importance of CT quality 
improvements to a broad audience. Change Teams will work with Implementation Teams 
(within the context of the RCT) to standardize practice, for example by showing the impact of 
organizational changes, creating disincentives for deviation, and providing monthly call access 
to physicists, technologists, or radiologists who can guide improvements. We will also regularly 
consult with our partners (including chief physicians, radiologists, and technologists) to 
understand what does and does not work and why. Close engagement of health care delivery 
staff as full partners will help to overcome critical implementation barriers. Examples of barriers 
might include CT systems that are too old to generate doses comparable to newer machines; or 
facilities that have so many protocols for imaging a single area that technologists cannot use 
them all in an optimized way; or that radiologists insist on multiphase or multi-sequence CT 
protocols without appreciating that those protocols are no longer considered state-of-the-art or 
understanding their impact on patient dose. These challenges each require different solutions 
and engagement of different individuals. Some facilities may be more resistant to change than 
others. We anticipate that comparative performance data combined with group assessment of 
the most effective strategies will be the strongest tools for local champions to encourage change. 
Thus, we will work with facility lead champions to collect and present appropriate dose metric 
data that will encourage action by senior hospital leadership. 
Many of the facilities do not have experience in conducting quality improvement activities. We 
have engaged the Institute for Clinical Systems Improvement (ICSI) to provide background 
education on quality improvement to help our facilities create appropriate expectations and 
change team members. 
Facilities receiving the multicomponent intervention will meet via WebEx online conferencing 

25 
 
August 29, 2017 
 
 
  
with other groups in their randomization cluster to share experiences (Quality Improvement 
calls) weekly. Change and Intervention Teams will participate in these calls. Each meeting will 
include physicists and radiologists who will answer technical questions and provide specific 
examples and strategies for effective dose management. Topics will include change strategies 
such as development and application of standardized procedures and protocols. Facilities will 
report concerns and barriers for group problem solving and guidance. Leaders in 
implementation sciences will provide concrete suggestions for changing practice. Shared data 
from the groups in the cluster will allow the best performers to describe how they achieved 
goals and give low performers concrete steps toward more effective action. Collaborative 
implementation is an ideal strategy for pooling experience. Educational materials will be 
offered and encouraged in these collaborative meetings. During each WebEx conference, we 
will keep track of the number of participants from each facility who attend, the number of 
questions that are asked by each facility or comments that are made, and the sharing of specific 
cases as a way to score the relative engagement of individuals from each facility in the quality 
improvement collaboratives. 
As with the audit intervention, the facilities will be responsible for determining how to use the 
provided advice to improve performance. 
M. Timeline and Timeline of Assessments 
The facilities will be divided into three tracks (i.e. randomization groups), each 
containing approximately 37 facility clusters.  Most facility clusters include a single 
facility, while some clusters will include several facilities that are integrated in how 
they perform CT.   Tracks will receive interventions in a sequentially staggered 
fashion, with their order determined randomly (see below). 
M.1. Graph of timeline (also included with attachments) 
 
 

26 
 
August 29, 2017 
 
 
  
 
 
 
 
 
 
M.2. Timing of assessments related to inventions 
Each facility’s baseline radiation dose levels will be assessed during a period of between 38 and 
52 weeks (9-12 months) prior to providing them with their initial simple audit. The timeline for 
baseline assessment will be November 2015 - July 2016 for the first randomization group (track), 
November 2015 - August 2016 for the second track, and November 2015  - October 2016 for the 
third track.  Post simple audit assessment will start 4 weeks after each facility receives its simple 
audit, and continue until the first multicomponent audit is shared (around January 2017).  We 
consider the 4-week period a washout period during which facilities can study the simple audit 
and begin to make changes in CT scanning operations that potentially could change CT 
radiation dose levels. This will create follow-up periods of approximately 9, 18, and 22 weeks 
(for tracks A, B, and C, respectively) for assessment of changes in dose following the simple 
audit. 
Pre-multicomponent dose assessment will begin in November 2016, 12 weeks prior to 
beginning of the first multicomponent intervention.  Assessment of changes in dose following 
the multicomponent intervention will begin 14 days after the final collaborative call (for each 
track), and end (for all tracks) in October 2017, approximately 12 weeks after the final 
collaborative call of the third track.  This will produce variable length post-multicomponent 
intervention assessment periods of approximately 10, 22, and 24 weeks (for tracks A, B, and C, 
respectively), during which dose metrics will be measured to assess changes. 
J F M A M J J A S O N D J F M A M J J A S O N D
week of the year
44
45
46
47
48
49
50
51
52
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
29
20
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
months
months
 study week
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
Simple Audit (S)
S
Multicomponent Intervention 
(MCI)
M
Surveys (Su)
Simple Audit (S)
S M
Multicomponent Intervention 
(MCI)
Surveys (Su)
Simple Audit (A)
S 
Multicomponent Intervention 
(B)
Surveys (Su)
S
Audit 
Surveys 
MCI
Multicomponent Intervention incuding 8 week collaboratives
M
In-person Meeting
Su
2019
November
December
January
February
January
September
October
November
December
March
April
May
June
July
August
Audit & Intervention
2015
2016
2017
2018
February
March
April
May
June
July
August
September
October
November December
Delayed 
Assessment  
after MCI and 
simple audit
Baseline Assesment: Before Simple Audit  
   Assessment: After Simple Audit
Baseline Assessment:      
Before Multicomponent 
Intervention
MCI
Baseline Su
Track 
C 
Baseline Assessment: Before Multicomponent 
Intervention
MCI
Assessent: After Multicomponent Intervention 
Track 
A     
Track 
B 
Interviews
Interviews
Assessment: After Multicomponent Intervention  
Baseline Assesment: Before Simple Audit
   Assessment: After Simple Audit  
Su1
Su2
Su2
Su2
MCI
Assessment: After 
Multicomponent Intervention
Assessment: After Simple 
Audit
Baseline Assessment: Before Multicomponent Intervention
Baseline Su
Baseline Su
Su1
Su1
Baseline Assesment: Before Simple Audit
Interviews
Delayed 
Assessment  
after MC and 
simple audit
Delayed 
Assessment  
after MCI and 
simple audit
Su3
Su3
Su3

27 
 
August 29, 2017 
 
 
  
In order to assess whether improvements in dose occur at or persist over a longer time period; 
we will compare doses 12-15 months after completion of the multicomponent intervention with 
baseline doses and doses after the simple audit and the multicomponent interventions. 
N. Statistical Analysis  
N.1 Quantitative Analyses 
For each trial phase, we will compare radiation doses from a baseline period before that 
intervention starts with doses in a post intervention period, separated by a “washout” period 
during which facilities should have adequate time to a) absorb the feedback and messages from 
the intervention, and b) implement changes in scanning practices if and how they so choose.  
The pre-audit/intervention and post-audit/intervention periods will vary in length of time 
according to: a) which type of intervention (simple or multicomponent), and b) which track 
(randomization group) the facilities are in.  Please see Section M for details.  
The primary outcome measure will be effective dose.  Secondary outcomes will be CTDIvol and 
DLP. For each anatomic area studied and each dose metric, we will compare the mean dose and 
the percentage of CT examinations with dose levels above the baseline 75th percentile 
benchmark threshold. Dose metrics may be transformed (e.g., by applying the natural 
logarithm) prior to modeling the mean dose if necessary to meet linear model assumptions. 
We will model mean doses (possibly transformed) using hierarchical linear regression and will 
model the percentage of doses above the threshold using hierarchical logistic regression, 
controlling for underlying time trends and for facility-level and patient-level covariates that 
could influence dose (e.g., facility size, patient age or sex, etc.). We will include random effects 
for machine, facility, and facility cluster to account for correlation among scans performed on 
the same machine, within the same facility, and/or within the same cluster of facilities. We will 
evaluate the effect of audit vs. no feedback on CT dose by testing whether dose levels decrease 
in the follow-up period after the audit vs. during the baseline period prior to the audit. We will 
use a similar approach for the multicomponent intervention but will include all periods in the 

28 
 
August 29, 2017 
 
 
  
model, comparing post intervention to both original (pre-simple audit/intervention) baseline 
and new (pre-multicomponent audit/intervention) baseline after completion of the final 
collaborative calls of the multicomponent audit/intervention. We will use a similar approach for 
assessing delayed and/or longer-term changes by analyzing doses 12-15 months following the 
multicomponent intervention in comparison to doses in the earlier periods.  
We will also compare the multicomponent intervention and simple audit effects to determine 
which is more effective. We will also compare how participation in the in-person meeting 
impacts the effects of the simple audit and multicomponent intervention by including 
interactions with an effect of in-person meeting and the particular intervention. Each facility 
will act as its own control, and we will control for underlying time trends, which are estimable 
because of the stepped-wedge design.  
N.2 Mixed methods qualitative and quantitative analyses 
The organizational survey includes a combination of quantitative and qualitative questions.  We 
will examine these questions using standard descriptive statistics. We will assess the 
relationship between facility-level characteristics as assessed through the surveys and dose 
levels (baselines as well as changes in dose), using responses to our facility surveys using mixed 
methods approaches. In order to make the analysis more robust we will construct domains 
using the survey questions. The first step in constructing these "domains" is to use exploratory 
factor analysis (EFA) to identify groups of questions that simultaneously correlate highly with 
each other but not to the point of redundancy.  We will then combine the questions within each 
of these groups, or domains, to form composite domain scores (i.e. measures) that will become 
continuous predictive variables, combined with structural facility-level variables from the first 
survey and with patient-level variables (e.g. age, size, etc.) for a series of linear models that 
assess how these facility- and patient-level measures relate to: a) baseline dose levels and b) 
changes in dose levels after the interventions in our study. 
Our final statistical models of dose and changes in dose will use primary (facility-level, scanner-
level, and patient-level) analytical variables derived directly from our Radimetrics database, in 

29 
 
August 29, 2017 
 
 
  
combination with facility-level characteristics derived from our two surveys.  We will use the 
same hierarchical modeling approaches as described above to estimate the relationships 
between our predictors (facility-level, scanner-level, patient-level, and various interventions) 
and dose measures. 
O. Sample Size  
Statistical power depends on between-class coefficient of variation (standard deviation between 
facilities divided by average prevalence per facility). Based on prior research, we expect these 
values to vary between 0.4 and 1.1, depending on anatomical region and response metric. With 
84 unique clusters, reflecting 116 facilities, to be randomized and at least 27-28 scans per facility 
for each anatomical region and age group (adult/children), in each measurement period, we will 
have an 80% power to detect a change of 5 percentage points (from 25% to 20%) in percentage of 
examinations with doses above the benchmark after the intervention versus baseline, using a 
two-sided significance level of 5%. The exceptions to this are children’s head scans, which only 
require 15 scans per facility in each measurement period to achieve the same power with 84 
clusters. We expect each facility to contribute, on average, at least 60 scans per day. With each 
measurement period being at least 1 month it is very likely 80% power will be surpassed. The 
effect of multicomponent intervention is expected to be greater than the effect of auditing; thus, 
the same sample size would also be sufficient in detecting the significance of multicomponent 
intervention. 
 
P. Ethical Considerations and DSMB 
The primary ethical concern for the project relates to loss of confidentiality for the participating 
facilities and institutions. We will do everything we can to maintain the confidentiality of the 
facilities. Through the project we may identify individuals who have received exceedingly high 
radiation exposures. These will be reviewed with the Data Safety Monitoring Board (DSMB) to 
determine what course of action, if any, is necessary. We will be interviewing staff at many of 
our facilities. Potentially they might disclose views that could be perceived as negative by their 

30 
 
August 29, 2017 
 
 
  
peers or supervisors. All of the survey results will be kept confidential and will not be reported 
back to the individual facilities so that the results cannot be linked to an individual or even to a 
facility or institution.  
Q. References 
1. 
Solberg LI. Improving medical practice: a conceptual framework. Ann Fam Med. May-Jun 
2007;5(3):251-256.  
2. 
Solberg LI, Asche SE, Margolis KL, Whitebird RR. Measuring an organization's ability to 
manage change: the  change process capability questionnaire and its use for improving 
depression care. American journal of medical quality: the official journal of the American 
College of Medical Quality. May-Jun 2008;23(3):193-200.  
3. 
Schindera ST, Treier R, von Allmen G, et al. An education and training programme for 
radiological institutes:  impact on the reduction of the CT radiation dose. European 
radiology. Oct 2011;21(10):2039-2045.  
4. 
Keegan J, Lane D, Miglioretti D, Gould R, Smith-Bindman R. Radiation dose metrics in 
computed tomography:  assessing dose using the National Quality Forum CT patient safety 
measure. In press, Journal of the American  Collge of Radiology. March 2014.  
5. 
Miglioretti DL, Zhang YX, E J, N V, R S-B. Personalized technologist dose audit feedback 
for reducing patient  radiation exposure from computed tomography. In press, Journal of 
the American College of Radiology. 2014.  
6. 
Pearce MS, Salotti JA, Little MP, et al. Radiation exposure from CT scans in childhood and 
subsequent risk of  leukaemia and brain tumours: a retrospective cohort study. Lancet. Aug 
4 2012;380(9840):499-505.  
7. 
Mathews J, Forsythe A, Brady Z, al. e. Cancer risk in 680  000 people exposed to computed 
tomography scans in  childhood or adolescence: data linkage study of 11 million 
Australians. BMJ. 2013;346 doi:  http://dx.doi.org/10.1136/bmj.f2360  
8. 
Conference of Radiation Control Program Directors. Nationwide evaluation of x-ray trends 
(NEXT): Tabulation  and graphical summary of 2000 survey of computed tomography. 

31 
 
August 29, 2017 
 
 
  
Frankfort, KY2007.  
9. 
Bhargavan M, Sunshine JH. Utilization of radiology services in the United States: levels and 
trends in modalities,  regions, and populations. Radiology. Mar 2005;234(3):824-832.  
10. 
Smith-Bindman R, Miglioretti D, Larson E. Rising Use of Diagnostic Medical Imaging in an 
Integrated Health  Plan. Health Affairs. 2008;26(6):1491-1502.  
11. 
Baker LC, Atlas SW, Afendulis CC. Expanded use of imaging technology and the challenge 
of measuring value.  Health Aff (Millwood). Nov-Dec 2008;27(6):1467-1478.  
12. 
Medicare Payment Advisory Commission. A Data Book: Healthcare Spending and the 
Medicare Program, June  2010. 2010.  
13. 
Levin DC, Rao VM, Parker L, Frangos AJ, Sunshine JH. Bending the curve: the recent 
marked slowdown in  growth of noninvasive diagnostic imaging. AJR. American journal of 
roentgenology. Jan 2011;196(1):W25-29.  
14. 
Smith-Bindman R, Miglioretti DL, Johnson E, et al. Use of diagnostic imaging studies and 
associated radiation  exposure for patients enrolled in large integrated health care systems, 
1996-2010. JAMA. Jun 13  2012;307(22):2400-2409.  
15. 
Mettler FA, Jr., Huda W, Yoshizumi TT, Mahesh M. Effective doses in radiology and 
diagnostic nuclear  medicine: a catalog. Radiology. Jul 2008;248(1):254-263.  
16. 
National Council on Radiation Protection and Measurements. NCRP Report No. 93, 
Ionizing Radiation Exposure  of the Population of the United States. 1987.  
17. 
National Council on Radiation Protection and Measurements. NCRP Report No 160, 
Ionizing Radiation  Exposure of the Population of the United States, press release available 
, full report soon to be released; 2009.  
18. 
Pierce DA, Preston DL. Radiation-related cancer risks at low doses among atomic bomb 
survivors. Radiat Res.  Aug 2000;154(2):178-186.  
19. 
Preston DL, Ron E, Tokuoka S, et al. Solid cancer incidence in atomic bomb survivors: 1958-
1998. Radiat Res.  Jul 2007;168(1):1-64.  
20. 
Brenner DJ, Doll R, Goodhead DT, et al. Cancer risks attributable to low doses of ionizing 

32 
 
August 29, 2017 
 
 
  
radiation: assessing  what we really know. Proc Natl Acad Sci U S A. Nov 25 
2003;100(24):13761-13766.  
21. 
Board of Radiation Effects Research Division on Earth and Life Sciences National Research 
Council of the  National Academies. Health Risks from Exposure to Low Levels of Ionizing 
Radiation: BEIR VII Phase 2  Washington, D.C.: The National Academies Press; 2006.  
22. 
Boutis K, Cogollo W, Fischer J, Freedman SB, Ben David G, Thomas KE. Parental 
knowledge of potential cancer  risks from exposure to computed tomography. Pediatrics. 
Aug 2013;132(2):305-311.  
23. 
Hertz-Picciotto I. Breast Cancer and the Environment: A Life Course Approach. Dec 2011 
2011.  
24. 
Brenner DJ, Elliston CD, Hall EJ, Berdon WE. Estimates of the cancer risks from pediatric 
CT radiation are not  merely theoretical: comment on "point/counterpoint: in x-ray 
computed tomography, technique factors should be selected appropriate to patient size. 
Against the proposition". Medical physics. Nov 2001;28(11):2387-2388.  
25. 
Berrington de Gonzalez A, Mahesh M, Kim KP, et al. Projected cancer risks from computed 
tomographic scans performed in the United States in 2007. Arch Intern Med. Dec 14 
2009;169(22):2071-2077.  
26. 
Berrington de Gonzalez A, Darby SC, . Risk of cancer from diagnostic x-rays: estiamtes for 
the UK and 14 other countries. The Lancet. 2004;363(9406):345-351.  
27. 
Smith-Bindman R, Lipson J, Marcus R, et al. Radiation dose associated with common 
computed tomography examinations and the associated lifetime attributable risk of cancer. 
Arch Intern Med. Dec 14 2009;169(22):2078- 2086.  
28. 
Rydberg J, Buckwalter KA, Caldemeyer KS, et al. Multisection CT: scanning techniques and 
clinical applications. Radiographics. Nov-Dec 2000;20(6):1787-1806.  
29. 
Halpern EJ. Triple-rule-out CT angiography for evaluation of acute chest pain and possible 
acute coronary syndrome. Radiology. Aug 2009;252(2):332-345.  
30. 
Sommer WH, Schenzle JC, Becker CR, et al. Saving dose in triple-rule-out computed 
tomography examination using a high-pitch dual spiral technique. Investigative radiology. 

33 
 
August 29, 2017 
 
 
  
Feb 2010;45(2):64-71.  
31. 
Foley WD, Mallisee TA, Hohenwalter MD, Wilson CR, Quiroz FA, Taylor AJ. Multiphase 
hepatic CT with a multirow detector CT scanner. AJR. American journal of roentgenology. 
Sep 2000;175(3):679-685.  
32. 
Roberts HC, Roberts TP, Smith WS, Lee TJ, Fischbein NJ, Dillon WP. Multisection dynamic 
CT perfusion for acute cerebral ischemia: the "toggling-table" technique. AJNR Am J 
Neuroradiol. Jun-Jul 2001;22(6):1077-1080.  
33. 
Hoeffner EG, Case I, Jain R, et al. Cerebral perfusion CT: technique and clinical 
applications. Radiology. Jun 2004;231(3):632-644.  
34. 
Smith AB, Dillon WP, Lau BC, et al. Radiation dose reduction strategy for CT protocols: 
successful implementation in neuroradiology section. Radiology. May 2008;247(2):499-506.  
35. 
McBride, Paxton B, Warddrop R. Majority of Ordering Physicians Lack Knowledge of 
Radiation Exposure Risks from CT. American Roentgen Ray Society, 2009, Boston MA. 
2009.  
36. 
Caolili EM, H.Cohan R, Ellis JH. Medical Decision Making Regarding Computed 
Tomographic Radiation Dose and Associated Risk: The Patient’s Perspective. Archives of 
Internal Medicine. 2009;169(11):1069-1071.  
37. 
Smith-Bindman R. Is computed tomography safe? N Engl J Med. Jul 1 2010;363(1):1-4.  
38. 
Duncan J, Evens R. Using Information to Optimize Medical Outcomes JAMA. 2009;301 
(2):383-2385.  
39. 
Poletti PA, Platon A, Rutschmann OT, Schmidlin FR, Iselin CE, Becker CD. Low-dose 
versus standard-dose CT  protocol in patients with clinically suspected renal colic. AJR. 
American journal of roentgenology. Apr  2007;188(4):927-933.  
40. 
Radiation protection 118 : referral guidelines for imaging. Luxembourg European 
Commission, Directorate-  General for the Environment; 2001.  
41. 
Eisenberg RL, Bankier AA, Boiselle PM. Compliance with Fleischner Society guidelines for 
management of  small lung nodules: a survey of 834 radiologists. Radiology. Apr 

34 
 
August 29, 2017 
 
 
  
2010;255(1):218-224.  
42. 
U.S. Food and Drug Administration. FDA White Paper: Initiative to Reduce Unnecessary 
Radiation Exposure  from Medical Imaging. 2010; http://www.fda.gov/Radiation- 
 EmittingProducts/RadiationSafety/RadiationDoseReduction/ucm199994.htm.  
43. 
US Food and Drug Administration. Safety Investigation of CT Brain Perfusion Scans: 
Update 11/9/2010. 2009.  
44. 
Bogdanich W. After stroke scans, patients face serious risk. New York TImes. July 31, 2010, 
2010.  
45. 
Bogdanich W. Listing of articles published in the NY TImes over the last year in the 
'Radiation Boom' series by  Walt Bogdanich examine issues arising from the increasing use 
of medical radiation and the new technologies  that deliver it. . 
http://topics.nytimes.com/top/news/us/series/radiation_boom/index.html?ref=health.  
46. 
The US House of Representatives, Energy and Commerce Committee, Subcommittee on 
Health held a hearing  entitled "Medical Radiation: An Overview of the Issues" on Friday, 
February 26, 2010, examining the potential benefits and risks of the use of radiation in 
medicine. Transcripts of testimony can be found at 
http://democrats.energycommerce.house.gov/index.php?q=hearing/medical-radiation-an-
overview-of-the-issues. 2010.  
47. 
Brody AS, Frush DP, Huda W, Brent RL. Radiation risk to children from computed 
tomography. Pediatrics. Sep 2007;120(3):677-682.  
48. 
UK National Health Program Department of Health. The Ionising Radiation (Medical 
Exposure) Regulations 2000 (together with notes on good practice). 2007, 
http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGu
idance/DH_4007957, accessed January 15, 2011.  
49. 
Wall BF. Diagnostic reference levels--the way forward. Br J Radiol. Sep 2001;74(885):785-
788.  
50. 
ICRP. 1990 recommendations of the International Commission on Radiological Protection, 
ICRP Publication 60. Annals of the ICRP 21, No. 1–3, 1991. 

35 
 
August 29, 2017 
 
 
  
51. 
ICRP. Radiological protection and safety in medicine, ICRP Publication 73. Annals of the 
ICRP 26, No. 2, 1996.  
52. 
Seeram E, Brennan PC. Diagnostic reference levels in radiology. Radiol Technol. May-Jun 
2006;77(5):373-384;  quiz 385-377.  
53. 
Amis ES, Jr., Butler PF, Applegate KE, et al. American College of Radiology white paper on 
radiation dose in  medicine. Journal of the American College of Radiology : JACR. May 
2007;4(5):272-284.  
54. 
Smith-Bindman R. Environmental causes of breast cancer and radiation from medical 
imaging: findings from the  Institute of Medicine report. Arch Intern Med. Jul 9 
2012;172(13):1023-1027.  
55. 
Center for Medicare and Medicaid Services. Section 135(b)of the Medicare Improvements 
for Patients and  Providers Act of 2008 (MIPPA) mandates an Appropriate Use of Imaging 
Services demonstration project. The goal of the demonstration is to collect data regarding 
physician use of advanced diagnostic imaging services to determine the appropriateness of 
services in relation to established criteria and physician peers. 2010; 
http://www.cms.gov/demoprojectsevalrpts/md/itemdetail.asp?itemid=CMS1222075.  
56. 
The Joint Commission Sentinel Event Alert System. 2011; 
http://www.jointcommission.org/assets/1/18/SEA_471.PDF.  
57. 
The US House of Representatives, Energy and Commerce Committee, Subcommittee on 
Health held a hearing entitled "Examining The Appropriateness Of Standards For Medical 
Imaging and Radiation Therapy Technologists" on June 8, 2012 examining standards for 
the conduct of medical imaging examinations and the Consistency, Accuracy, 
Responsibility and Excellence in Medical Imaging and Radiation Therapy Bill (H.R. 2104, 
C.A.R.E). Transcripts of Dr. Smith-Bindman's testimony can be found at 
http://democrats.energycommerce.house.gov/sites/default/files/image_uploads/Testiomon
y_Health_06.08.12_Smi thBindman.pdf.  
58. 
Image Gently. 2009. http://www.pedrad.org/associations/5364/ig/index.cfm?page=369. 
Accessed Feb 13, 2009.  

36 
 
August 29, 2017 
 
 
  
59. 
Catalano C, Francone M, Ascarelli A, Mangia M, Iacucci I, Passariello R. Optimizing 
radiation dose and image  quality. European radiology. Dec 2007;17 Suppl 6:F26-32.  
60. 
Lee CI, Haims AH, Monico EP, Brink JA, Forman HP. Diagnostic CT scans: assessment of 
patient, physician,  and radiologist awareness of radiation dose and possible risks. 
Radiology. May 2004;231(2):393-398.  
61. 
Damschroder LJ, Banaszak-Holl J, Kowalski CP, Forman J, Saint S, Krein SL. The role of the 
champion in  infection prevention: results from a multisite qualitative study. Qual Saf 
Health Care. Dec 2009;18(6):434-440.  
62. 
Miglioretti DL, Zhang Y, Johnson E, Vanneman N, Smith-Bindman R. Personalized 
Technologist Dose Audit  Feedback for Reducing Patient Radiation Exposure from 
Computed Tomography. submitted for publication 2013.  
63. 
Aarons G, Horowitz J, Dlugosz L, Ehrhart M. The role of organizational processes in 
dissemination and  implementation researach. In: Brownson RC, Colditz GA, Proctor EK, 
eds. Dissemination and Implementation  Research in Health: Translating Science to 
Practice. New York: Oxford University Press, Inc.; 2012:128-153.  
64. 
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering 
implementation of health  services research findings into practice: a consolidated 
framework for advancing implementation science.  Implement Sci. 2009;4:50.  
65. 
Solberg LI, Asche SE, Margolis KL, Whitebird RR, Trangle MA, Wineman AP. Relationship 
between the  presence of practice systems and the quality of care for depression. Am J Med 
Qual. Nov-Dec 2008;23(6):420-  426.  
66. 
Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the 
impact of health  promotion programs: using the RE-AIM framework to form summary 
measures for decision making involving  complex issues. Health education research. Oct 
2006;21(5):688-694.  
67. 
Glasgow RE, Dickinson P, Fisher L, et al. Use of RE-AIM to develop a multi-media 
facilitation tool for the  patient-centered medical home. Implement Sci. 2011;6:118.  
68. 
Fortney J, Enderle M, McDougall S, et al. Implementation outcomes of evidence-based 

37 
 
August 29, 2017 
 
 
  
quality improvement for  depression in VA community based outpatient clinics. 
Implement Sci. 2012;7:30.  
69. 
Glasgow RE. RE-AIMing research for application: ways to improve evidence for family 
medicine. Journal of the  American Board of Family Medicine : JABFM. Jan-Feb 
2006;19(1):11-19.  
70. 
Cho RC, Chu P, Smith-Bindman R. Second trimester prenatal ultrasound for the detection 
of pregnancies at  increased risk of Trisomy 18 based on serum screening. Prenat Diagn. 
Feb 2009;29(2):129-139.  
71. 
Haggstrom DA, Quale C, Smith-Bindman R. Differences in the quality of breast cancer care 
among vulnerable  populations. Cancer. Dec 1 2005;104(11):2347-2358.  
72. 
Kagay C, Quale C, Smith-Bindman R. Screening Mammography in The American Elderly. 
3. 2006;31(2):142-  149.  
73. 
Kerlikowske K, Smith-Bindman R, Abraham LA, et al. Breast cancer yield for screening 
mammographic examinations with recommendation for short-interval follow-up. 
Radiology. Mar 2005;234(3):684-692. 
74. 
Levine D, Brown DL, Andreotti RF, et al. Management of asymptomatic ovarian and other 
adnexal cysts imaged at US: Society of Radiologists in Ultrasound consensus conference 
statement. Ultrasound Q. Sep 2010;26(3):121- 131.  
75. 
Miglioretti DL, Smith-Bindman R, Abraham L, et al. Radiologist characteristics associated 
with interpretive performance of diagnostic mammography. J Natl Cancer Inst. Dec 19 
2007;99(24):1854-1863.  
76. 
Schueler K, Chu P, Smith-Bindman R. “Racial Differences in the Use of Mammography:” A 
Meta-analytical Review of Literature Describing Economic, Social, and Cultural Reasons 
for Variation in Use of Mammography”. Submitted. 2005.  
77. 
Sickles EA, Miglioretti DL, Ballard-Barbash R, et al. Performance benchmarks for 
diagnostic mammography. Radiology. Jun 2005;235(3):775-790.  
78. 
Smith-Bindman R, Ballard-Barbash R, Miglioretti DL, Patnick J, Kerlikowske K. Comparing 
the performance of mammography screening in the USA and the UK. J Med Screen. 

38 
 
August 29, 2017 
 
 
  
2005;12(1):50-54.  
79. 
Smith-Bindman R, Chu P, Bacchetti P, Waters JJ, Mutton D, Alberman E. Prenatal screening 
for Down syndrome in England and Wales and population-based birth outcomes. Am J 
Obstet Gynecol. Oct 2003;189(4):980-985.  
80. 
Smith-Bindman R, Chu P, Goldberg JD. Second trimester prenatal ultrasound for the 
detection of pregnancies at increased risk of Down syndrome. Prenat Diagn. Jun 
2007;27(6):535-544.  
81. 
Smith-Bindman R, Chu P, Miglioretti DL, et al. Physician predictors of mammographic 
accuracy. J Natl Cancer Inst. Mar 2 2005;97(5):358-367.  
82. 
Smith-Bindman R, Chu PW, Ecker J, Feldstein VA, Filly RA, Bacchetti P. Adverse birth 
outcomes in relation to prenatal sonographic measurements of fetal size. J Ultrasound Med. 
Apr 2003;22(4):347-356; quiz 357-348.  
83. 
Smith-Bindman R, Chu PW, Ecker JL, Feldstein VA, Filly RA, Bacchetti P. US evaluation of 
fetal growth: prediction of neonatal outcomes. Radiology. Apr 2002;223(1):153-161.  
84. 
Smith-Bindman R, Chu PW, Miglioretti DL, et al. Comparison of screening mammography 
in the United States and the United kingdom. Jama. Oct 22 2003;290(16):2129-2137.  
85. 
Smith-Bindman R, Hosmer W, Feldstein VA, Deeks JJ, Goldberg JD. Second-trimester 
ultrasound to detect fetuses with Down syndrome: a meta-analysis. Jama. Feb 28 
2001;285(8):1044-1055.  
86. 
Smith-Bindman R, Hosmer WD, Caponigro M, Cunningham G. The variability in the 
interpretation of prenatal diagnostic ultrasound. Ultrasound Obstet Gynecol. Apr 
2001;17(4):326-332.  
87. 
Smith-Bindman R, Kerlikowske K, Feldstein VA, et al. Endovaginal ultrasound to exclude 
endometrial cancer and other endometrial abnormalities. Jama. Nov 4 1998;280(17):1510-
1517.  
88. 
Smith-Bindman R, Kerlikowske K, Gebretsadik T, Newman J. Is screening mammography 
effective in elderly women? Am J Med. Feb 2000;108(2):112-119.  

39 
 
August 29, 2017 
 
 
  
89. 
Smith-Bindman R, McCulloch CE, Ding A, Quale C, Chu PW. Diagnostic imaging rates for 
head injury in the ED and states' medical malpractice tort reforms. Am J Emerg Med. Jul 12 
2010.  
90. 
Smith-Bindman R, Weiss E, Feldstein V. How thick is too thick? When endometrial 
thickness should prompt biopsy in postmenopausal women without vaginal bleeding. 
Ultrasound Obstet Gynecol. Oct 2004;24(5):558-565.  
91. 
Stengel JW, Webb EM, Poder L, Yeh BM, Smith-Bindman R, Coakley FV. Acute 
appendicitis: clinical outcome in patients with an initial false-positive CT diagnosis. 
Radiology. Jul 2010;256(1):119-126.  
92. 
Venkatesan A, Chu P, Kerlikowske K, Sickles EA, Smith-Bindman R. Positive predictive 
value of specific mammographic findings according to reader and patient variables. 
Radiology. Mar 2009;250(3):648-657.  
93. 
Goldstein RB, Bree RL, Benson CB, et al. Evaluation of the woman with postmenopausal 
bleeding: Society of Radiologists in Ultrasound-Sponsored Consensus Conference 
statement. J Ultrasound Med. Oct 2001;20(10):1025-1036.  
94. 
Solberg LI, Vinz C, Trevis JE. A technology solution for the high-tech diagnostic imaging 
conundrum. Am J Manag Care. Aug 2012;18(8):421-425.  
95. 
Lauren Crain A, Solberg LI, Unutzer J, et al. Designing and Implementing Research on a 
Statewide Quality Improvement Initiative: The DIAMOND Study and Initiative. Med Care. 
Feb 8 2012.  
96. 
Boyle R, Solberg L, Fiore M. Use of electronic health records to support smoking cessation. 
Cochrane Database Syst Rev. 2011(12):CD008743.  
97. 
Solberg LI. Care coordination: what is it, what are its effects and can it be sustained? Fam 
Pract. Oct 2011;28(5):469-470.  
98. 
Fontaine P, Flottemesch TJ, Solberg LI, Asche SE. Is consistent primary care within a 
patient-centered medical home related to utilization patterns and costs? J Ambul Care 
Manage. Jan-Mar 2011;34(1):10-19.  
99. 
Solberg LI, Glasgow RE, Unutzer J, et al. Partnership research: a practical trial design for 

40 
 
August 29, 2017 
 
 
  
evaluation of a natural experiment to improve depression care. Medical care. Jul 
2010;48(7):576-582.  
100. Whitebird RR, Solberg LI, Margolis KL, Asche SE, Trangle MA, Wineman AP. Barriers to 
improving primary care of depression: perspectives of medical group leaders. Qualitative 
health research. Jun 2013;23(6):805-814.  
101. Whitebird RR, Kreitzer M, Crain AL, Lewis BA, Hanson LR, Enstad CJ. Mindfulness-based 
stress reduction for family caregivers: a randomized controlled trial. The Gerontologist. 
Aug 2013;53(4):676-686.  
102. Jilcott S, Ammerman A, Sommers J, Glasgow RE. Applying the RE-AIM framework to 
assess the public health impact of policy change. Annals of behavioral medicine : a 
publication of the Society of Behavioral Medicine. Oct 2007;34(2):105-114.  
103. Whitebird RR, Kreitzer MJ, Lewis BA, et al. Recruiting and retaining family caregivers to a 
randomized controlled trial on mindfulness-based stress reduction. Contemporary clinical 
trials. Sep 2011;32(5):654-661.  
104. Whitebird RR, Kreitzer MJ, O'Connor PJ. Mindfulness-Based Stress Reduction and 
Diabetes. Diabetes spectrum : a publication of the American Diabetes Association. Sep 21 
2009;22(4):226-230.  
105. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over 
time. American journal of public health. Jun 2013;103(6):e38-46.  
106. Boone JM, Hendee WR, McNitt-Gray MF, Seltzer SE. Radiation exposure from CT scans: 
how to close our knowledge gaps, monitor and safeguard exposure--proceedings and 
recommendations of the Radiation Dose Summit, sponsored by NIBIB, February 24-25, 
2011. Radiology. Nov 2012;265(2):544-554.  
107. Boone JM. Method for evaluating bow tie filter angle-dependent attenuation in CT: theory 
and simulation results. Medical physics. Jan 2010;37(1):40-48.  
108. Boone JM. Dose spread functions in computed tomography: a Monte Carlo study. Medical 
physics. Oct 2009;36(10):4547-4554.  
109. Huda W, Nickoloff EL, Boone JM. Overview of patient dosimetry in diagnostic radiology in 

41 
 
August 29, 2017 
 
 
  
the USA for the past 50 years. Medical physics. Dec 2008;35(12):5713-5728.  
110. Boone JM, Brunberg JA. Computed tomography use in a tertiary care university hospital. 
Journal of the American College of Radiology : JACR. Feb 2008;5(2):132-138.  
111. Boone JM. Multidetector CT: opportunities, challenges, and concerns associated with 
scanners with 64 or more detector rows. Radiology. Nov 2006;241(2):334-337.  
112. Boone JM, Kwan AL, Seibert JA, Shah N, Lindfors KK, Nelson TR. Technique factors and 
their relationship to radiation dose in pendant geometry breast CT. Medical physics. Dec 
2005;32(12):3767-3776.  
113. Schindera ST, Nelson RC, Lee ER, et al. Abdominal multislice CT for obese patients: effect 
on image quality and radiation dose in a phantom study. Academic radiology. Apr 
2007;14(4):486-494.  
114. Schindera ST, Nelson RC, Toth TL, et al. Effect of patient size on radiation dose for 
abdominal MDCT with automatic tube current modulation: phantom study. AJR. 
American journal of roentgenology. Feb 2008;190(2):W100-105.  
115. Szucs-Farkas Z, Kurmann L, Strautz T, Patak MA, Vock P, Schindera ST. Patient exposure 
and image quality of low-dose pulmonary computed tomography angiography: 
comparison of 100- and 80-kVp protocols. Investigative radiology. Dec 2008;43(12):871-876.  
116. Schindera ST, Nelson RC, Yoshizumi T, et al. Effect of automatic tube current modulation 
on radiation dose and image quality for low tube voltage multidetector row CT 
angiography: phantom study. Academic radiology. Aug 2009;16(8):997-1002.  
117. Szucs-Farkas Z, Schaller C, Bensler S, Patak MA, Vock P, Schindera ST. Detection of 
pulmonary emboli with CT angiography at reduced radiation exposure and contrast 
material volume: comparison of 80 kVp and 120 kVp protocols in a matched cohort. 
Investigative radiology. Dec 2009;44(12):793-799.  
118. Szucs-Farkas Z, Strautz T, Patak MA, Kurmann L, Vock P, Schindera ST. Is body weight the 
most appropriate criterion to select patients eligible for low-dose pulmonary CT 
angiography? Analysis of objective and subjective image quality at 80 kVp in 100 patients. 
European radiology. Aug 2009;19(8):1914-1922.  

42 
 
August 29, 2017 
 
 
  
119. Marin D, Nelson RC, Schindera ST, et al. Low-tube-voltage, high-tube-current 
multidetector abdominal CT: improved image quality and decreased radiation dose with 
adaptive statistical iterative reconstruction algorithm-- initial clinical experience. 
Radiology. Jan 2010;254(1):145-153.  
120. Schindera ST, Nauer C, Treier R, et al. [Strategies for reducing the CT radiation dose]. Der 
Radiologe. Dec 2010;50(12):1120, 1122-1127.  
121. Marin D, Nelson RC, Guerrisi A, et al. 64-section multidetector CT of the upper abdomen: 
optimization of a saline chaser injection protocol for improved vascular and parenchymal 
contrast enhancement. European radiology. Sep 2011;21(9):1938-1947.  
122. Marin D, Nelson RC, Rubin GD, Schindera ST. Body CT: technical advances for improving 
safety. AJR. American journal of roentgenology. Jul 2011;197(1):33-41.  
123. Schindera ST, Diedrichsen L, Muller HC, et al. Iterative reconstruction algorithm for 
abdominal multidetector CT at different tube voltages: assessment of diagnostic accuracy, 
image quality, and radiation dose in a phantom study. Radiology. Aug 2011;260(2):454-462.  
124. Szucs-Farkas Z, Schibler F, Cullmann J, et al. Diagnostic accuracy of pulmonary CT 
angiography at low tube voltage: intraindividual comparison of a normal-dose protocol at 
120 kVp and a low-dose protocol at 80 kVp using reduced amount of contrast medium in a 
simulation study. AJR. American journal of roentgenology. Nov 2011;197(5):W852-859.  
125. Schindera ST, Odedra D, Raza SA, et al. Iterative Reconstruction Algorithm for CT: Can 
Radiation Dose Be Decreased while Low-Contrast Detectability Is Preserved? Radiology. 
Jun 20 2013.  
126. Schindera ST, Winklehner A, Alkadhi H, et al. Effect of automatic tube voltage selection on 
image quality and radiation dose in abdominal CT angiography of various body sizes: a 
phantom study. Clinical radiology. Feb 2013;68(2):e79-86.  
127. Loupatatzis C, Schindera S, Gralla J, et al. Whole-body computed tomography for multiple 
traumas using a triphasic injection protocol. European radiology. Jun 2008;18(6):1206-1214.  
128. Solberg L. Quality improvement in primary care: The role of organization, systems, and 
collaboratives. In: Sollecito WA, Johnson JK, eds. McLaughlin and Kaluzny's continuous 

43 
 
August 29, 2017 
 
 
  
quality improvement in health care. Fourth ed. Burlington, MA: Jones & Bartlett Learning; 
399-419. 2011.  
129. Solberg LI, Brekke ML, Fazio CJ, et al. Lessons from experienced guideline implementers: 
attend to many factors and use multiple strategies. The Joint Commission journal on 
quality improvement. Apr 2000;26(4):171-188.  
130. Solberg LI, Kottke TE, Brekke ML, Magnan S. Improving prevention is difficult. Effective 
clinical practice :  ECP. May-Jun 2000;3(3):153-155.  
131. Solberg LI, Kottke TE, Brekke ML, et al. Failure of a continuous quality improvement 
intervention to increase the  delivery of preventive services. A randomized trial. Effective 
clinical practice : ECP. May-Jun 2000;3(3):105-  115.  
132. Solberg LI, Hroscikoski MC, Sperl-Hillen JM, O'Connor PJ, Crabtree BF. Key issues in 
transforming health care  organizations for quality: the case of advanced access. Joint 
Commission journal on quality and safety. Jan  2004;30(1):15-24.  
133. Hroscikoski MC, Solberg LI, Sperl-Hillen JM, Harper PG, McGrail MP, Crabtree BF. 
Challenges of change: a  qualitative study of chronic care model implementation. Annals of 
family medicine. Jul-Aug 2006;4(4):317-326.  
134. Solberg LI, Crain AL, Sperl-Hillen JM, Hroscikoski MC, Engebretson KI, O'Connor PJ. Care 
quality and  implementation of the chronic care model: a quantitative study. Annals of 
family medicine. Jul-Aug  2006;4(4):310-316.  
135. Solberg LI, Engebretson KI, Sperl-Hillen JM, O'Connor PJ, Hroscikoski MC, Crain AL. 
Ambulatory care quality  measures for the 6 aims from administrative data. American 
journal of medical quality : the official journal of the  American College of Medical Quality. 
Sep-Oct 2006;21(5):310-316.  
136. Solberg LI, Taylor N, Conway WA, Hiatt RA. Large multispecialty group practices and 
quality improvement:  what is needed to transform care? The Journal of ambulatory care 
management. Jan-Mar 2007;30(1):9-17;  discussion 18-20.  
137. American Association of Physicists in Medicine Position Statement on Radiation Risks from 
Medical Imaging  Procedures. 2011.  

44 
 
August 29, 2017 
 
 
  
138. Huda W, Magill D, He W. CT effective dose per dose length product using ICRP 103 
weighting factors. Medical  physics. Mar 2011;38(3):1261-1265.  
139. ICRP. ICRP Publication 103: Recommendations of the ICRP: Annals of the ICRP Volume 
37/2-4, 1e  (International Commission on Radiological Protection) (v. 37/2-4).  
140. Land CE. Studies of cancer and radiation dose among atomic bomb survivors. The example 
of breast cancer.  JAMA. Aug 2 1995;274(5):402-407.  
141. Linet M, Kim K, Rajaraman P. Children’s Exposure to Diagnostic Medical Radiation and 
Cancer Risk:  Epidemiologic and Dosimetric Considerations Pediatr Radiol 
2009;39((S1)):S4-S26.  
142. Brenner D, Elliston C, Hall E, Berdon W. Estimated risks of radiation-induced fatal cancer 
from pediatric CT.  AJR. American journal of roentgenology. Feb 2001;176(2):289-296.  
143. FDA Public Health Notification: Reducing Radiation Risk from Computed Tomography for 
Pediatric and Small  Adult Patients. 2001. Accessed January 18, 2007.  
144. Brenner DJ, Hall EJ. Computed tomography--an increasing source of radiation exposure. N 
Engl J Med. Nov 29  2007;357(22):2277-2284.  
145. Board of Radiation Effects Research Division on Earth and Life Sciences National Research 
Council of the National Academies. Committee on the Biological Effects of Ionizing 
Radiation, Health Effects of Exposure to Low Levels of Ionizing Radiation: BEIR V, 
Washington, DC: National Academy Press, 1990.  
146. Paterson A, Frush DP, Donnelly LF. Helical CT of the body: are settings adjusted for 
pediatric patients? AJR. American journal of roentgenology. Feb 2001;176(2):297-301.  
147. American Association of Physicists in Medicine. AAPM Report No 96. The Measurement, 
reporting and management of radiation dose in CT, Report of AAPM Task group 23 of the 
Diagnostic Imaging Council CT Committee. College Park, MD: American Association of 
Physicists in Medicine; 2008.  
148. NQF. National Quality Forum Measure 0739, Radiation Dose of Computed Tomography 
(CT) Steward: University of California San Francisco. 2011; 
http://www.qualityforum.org/QPS/0739  

45 
 
August 29, 2017 
 
 
  
149. Mittman B. Implementation Science in Health Care. In: Brownson RC, Colditz GA, Proctor 
EK, eds. Dissemination and Implementation Research in Health. New York: Oxford 
University Press; 2012:400-418.  
150. Solberg LI. Guideline implementation: what the literature doesn't tell us. The Joint 
Commission journal on quality improvement. Sep 2000;26(9):525-537.  
151. Chaillet N, Dube E, Dugas M, al. e. Evidence-based strategies for implementing guidelines 
in obstetrics: a systematic review. . Obstet Gynecol. . 2006;108(5):1234-1245.  
152. Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, AD. O. Does telling people what 
they have been doing change what they do? A systematic review of the effects of audit and 
feedback Qual Saf Health Care. 2006;15(6):433-436.  
153. Shojania KG, Ranji SR, McDonald KM, al. e. Effects of quality improvement strategies for 
type 2 diabetes on glycemic control: a meta-regression analysis JAMA. 2006;296 (4):427-440.  
154. Prior M, Guerin M, K. G-S. The effectiveness of clinical guideline implementation 
strategies--a synthesis of systematic review findings. . J Eval Clin Pract. 2008;14(5):888-897.  
155. SJ. H. Meta-analysis: audit and feedback features impact effectiveness on care quality. . 
Medical care. 2009;47(3):356-363.  
156. Brouwers MC, De Vito C, Bahirathan L, al. e. What implementation interventions increase 
cancer screening rates? a systematic review Implement Sci. 2011;6(1):111.  
157. Grimshaw J, Eccles M, Thomas R, et al. Toward evidence-based quality improvement. 
Evidence (and its limitations) of the effectiveness of guideline dissemination and 
implementation strategies 1966-1998. Journal of general internal medicine. Feb 2006;21 
Suppl 2:S14-20.  
158. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Medical 
Research Methodology. 2006;6(54).  
159. Solberg LI, Brekke ML, Fazio CJ, et al. Lessons from experienced guideline implementers: 
attend to many factors and use multiple strategies. The Joint Commission journal on 
quality improvement. Apr 2000;26(4):171-188.  

46 
 
August 29, 2017 
 
 
  
160. Stone EG, Morton SC, Hulscher ME, et al. Interventions that increase use of adult 
immunization and cancer screening services: a meta-analysis. Annals of internal medicine. 
May 7 2002;136(9):641-651.  
161. Solberg LI, Asche SE, Margolis KL, Whitebird RR. Measuring an organization's ability to 
manage change: the change process capability questionnaire and its use for improving 
depression care. American journal of medical quality : the official journal of the American 
College of Medical Quality. May-Jun 2008;23(3):193-200.  
162. Glasgow RE. RE-AIMing research for application: ways to improve evidence for family 
medicine. Journal of the American Board of Family Medicine : JABFM. Jan-Feb 
2006;19(1):11-19.  
163. Leconte I, Feger C, Galant C, et al. Mammography and subsequent whole-breast 
sonography of nonpalpable breast cancers: the importance of radiologic breast density. 
AJR. American journal of roentgenology. Jun 2003;180(6):1675-1679.  
164. Marshall C, & Rossman, G.B. Designing Qualitative Research. 4th ed. Newbury Park: Sage 
Publications; 2006.  
165. Huberman MM. Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, 
CA: Sage Publications;  1994.  
166. Patton MQ. Qualitative Research and Evaluation Methods. 3rd ed. Newbury Park: Sage 
Publications; 2002.  
167. Padgett D. Qualitative Methods in Social Work Research: Challenges and Rewards. 
Thousand Oaks, CA: Sage;  1998.  
168. Seale C. The Quality of Qualitative Research. Thousand Oaks: Sage Publications; 1999.  
169. Greevy R, Lu B, Silber JH, Rosenbaum P. Optimal multivariate matching before 
randomization. Biostatistics 2004;5:263-75 
170. Greevy RA, Jr., Grijalva CG, Roumie CL, et al. Reweighted Mahalanobis distance matching 
for cluster-randomized trials with missing data. Pharmacoepidemiol Drug Saf 2012;21 
Suppl 2:148-54. 

47 
 
August 29, 2017 
 
 
  
171. Morgan KL, Rubin, D.L. Rerandomization to improve covariate balance in experiments. 
The Annals of Statistics 2012;40:1263–82. 
 
 
 

48 
 
August 29, 2017 
 
 
  
R. APPENDICES 
 
R.1. Variable List - Survey 
Facility-Level Variables 
Organization Location 
  
 
 
 
 
Geographic location (U.S., non U.S.; Geographic region of the U.S.) 
 
Population size of community where facility located 
Organization Structure 
 
Services provided (hospital/inpatient, emergency, ambulatory, other)  
 
Type of Facility 
 
 
 
Academic/Teaching Hospital 
Trauma Center (Level 1, 2, or 3) 
Public Hospital 
Community Hospital 
Private Hospital 
Acute Care Facility 
Primary Cancer Facility 
Pediatric Hospital 
Tertiary Referral Hospital 
Outpatient Imaging Facility 
 
Payer mix 
 
Racial and ethnic breakdown of patients 
Staffing and Capacity 
 
Number of radiologists 
 
Number of technologists 
 
Number of CT machines 
 
Number of medical physicists 
CT Protocols (instructions) 
 
Number by anatomic area 

49 
 
August 29, 2017 
 
 
  
 
Who establishes them, who can modify them 
Manufacturer (protocols pre-set on machines) 
Organizational leadership 
Medical Physicist (one for the org.) 
Medical Physicist (at the particular hospital) 
Radiology site 
Lead radiologists 
Any individual radiologists 
Head Technologist 
Technologist performing exams 
Other  
 
Are protocols locked 
  
 
  
 
 
How often are protocols reviewed and updated 
Facility Practice Sharing 
  
 
 
 
Activities focused on optimization 
Status of ongoing efforts on optimization 
 
 
 
 
 

50 
 
August 29, 2017 
 
 
  
R.2. Variable List – Dose Registry Variables 
 
Patient-Level Variables 
Sex 
Age 
Size (mid scan diameter) 
  
 
 
 
Indication for imaging (derived) 
  
 
 
 
 
CT examination-level variables (each examination/encounter may include 1 or more radiating 
events/ imaging acquisitions) 
Time of examination (date, day of the week, time of the day) 
Machine (make, model) 
Study description 
Protocol name 
Scan region(s):  
Dose parameters of each imaging acquisition 
 
Average mAs  
 
Average kVp 
 
Scan length 
 
Pitch 
 
Collimation 
 
Scan thickness 
  
  
  
 
CTDIvol for each imaging acquisition 
DLP for each imaging acquisition 
Effective Dose for each imaging acquisition 
CTDIvol (average, weighted by scan length) for each examination 
DLP (sum) for each examination  
Effective Dose (sum) for each examination 
 

51 
 
August 29, 2017 
 
 
  
R.3 Parental Organizational Survey (pdf attached) 
R.4 Sublevel Organizational Survey (pdf attached) 
R.5 Implementation Survey (pdf attached) 
R.6 Timeline (pdf attached)