Improving Communication Between Cancer Patients and Oncologists Using Patient Feedback on Actual Conversations and the ABIM Maintenance of Certification Program
Phase: NA
Conditions: Cancer
Sponsor: Dana-Farber Cancer Institute

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
 
 
 
 
 
Improving Communication Between Cancer Patients & Oncologists 
Protocol 
 
NCT#: 02969031 
 
05/07/2018 
 
 
 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
 
Improving Communication Between Cancer Patients and Oncologists Using Patient Feedback on Actual 
Conversations and the ABIM Maintenance of Certification Program. 
James Tulsky, MD 
Dana-Farber Cancer Institute 
Table of Contents 
1.0 INTRODUCTION...........................................................................................................................1 
1.1 Overview..............................................................................................................................1 
1.2 Background and Rationale...................................................................................................1 
 
2.0 AIMS...............................................................................................................................................7 
 
3.0 RESEARCH SUBJECT SELECTION........................................................................................7 
 
4.0 RESEARCH SUBJECT ENTRY.................................................................................................8 
 
5.0 STUDY DESIGN AND METHODS..........................................................................................10 
5.1 Design/Study Type............................................................................................................10 
5.2 Selection of Instruments....................................................................................................10 
5.3 Description of Intervention................................................................................................12 
5.4 Data Collection..................................................................................................................16 
5.5 Description of Study Processes.........................................................................................18 
5.5.1 Instrument Administration................................................................................18 
5.5.2 Intervention Administration..............................................................................18 
5.5.3 Special Concerns...............................................................................................19 
5.5.4 Compensation....................................................................................................19 
5.6 Adverse Reactions and Their Management.......................................................................19 
5.6.1 Reporting Adverse or Unanticipated Events.....................................................19 
5.6.2 Anticipated Reactions........................................................................................19 
5.6.3 Reaction Management.......................................................................................19 
 
6.0 STATISTICAL ANALYSIS........................................................................................................19 
6.1 Primary and Secondary Aims.............................................................................................19 
6.2 Sample Size and Statistical Power......................................................................................20 
6.3 Stratification Factors and Intervention Allocation Plan for Randomized Studies..............22 
6.4 Stratification Factors and their Impact on Design..............................................................23 
6.5 Early Stopping Rules, if appropriate..................................................................................23 
6.6 Definition of and Allowance in Design for Unevaluable/Ineligible Participants...............23 
6.7 Analysis Plan......................................................................................................................23 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
 
6.8 Handling of Missing Data in Analysis................................................................................25 
6.9 Multiple Testing for Subgroup and Exploratory Analysis..................................................26 
 
7.0 REFERENCES..............................................................................................................................27 
 
8.0 APPENDICES 
 
Appendix A: Patient Flyer 
Appendix B: Patient Survey 
Appendix C: Oncologist Recruitment Materials 
Appendix D: Oncologist Baseline Questionnaire 
Appendix E: ABIM Sampling Guidelines 
Appendix F: PCORI Oncologist Post-Participation Survey 
Appendix G: Information Sheet for Patients to be Recorded 
Appendix H: Information Sheet for Caregivers to be Recorded 
 
  
 
 
 
 
 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
1 
 
1.0 INTRODUCTION  
 
1.1 Overview 
Cancer patients face numerous physical and emotional challenges. Anxiety and depression can decrease 
quality of life and even decrease survival. High quality cancer care depends upon recognizing and 
responding to patients’ emotional needs. Unfortunately, oncologists frequently do not respond to 
patients’ emotional concerns. 
Oncologists can learn to communicate better. The most successful teaching occurs in intensive courses 
with patient actors. However, these courses take time and are expensive. We created a computerized 
program called SCOPE, Study of Communication in Oncologist Patient Encounters, in which 
oncologists audio record conversations with real patients and then receive feedback on how well they 
did. Oncologists who used this program in a research study doubled how often they responded to 
patients’ emotional concerns and increased their patients’ trust. 
We will take this program and add an additional feature in which real patients also listen to the 
conversations and give feedback to these doctors. We will then have doctors who are being recertified in 
oncology by the American Board of Internal Medicine use this program as part of that recertification 
process. We will test how well it increases their patients’ satisfaction in their communication, and 
whether it improves their communication skills. 
If successful, this intervention will represent an effective communication training program for physicians 
in which they improve their communication skills through direct feedback from patients. Because of the 
integration with the ABIM Maintenance of Certification Program, if fully adopted into that program, it 
could have an extraordinary impact on thousands of physicians and their patients. 
 
1.2 Background and Rationale: 
People living with cancer suffer greatly and the burden is greatest on those with advanced disease.  
Cancer patients experience extraordinary physical, emotional, and existential challenges, and up to 60% 
acknowledge difficulty coping with their illness.1-3  Among patients with advanced disease, 25% describe 
their suffering as moderate to extreme.4 Physical symptoms are the greatest source of this distress, and up 
to 70% of patients with advanced cancer experience pain severe enough to impair function.5-7  Emotional 
suffering can be profound as well, as patients experience fear, anger, sadness or isolation.  Psychological 
distress, particularly anxiety or depression, is common in all cancer patients, with a reported prevalence 
between 30-50%.2,8-13 Psychological and emotional distress has been correlated with lower quality of 
life,13-16 a desire to hasten death,17 caregiver distress,18-20 increased health care utilization21-23 and shorter 
survival.24-26  Facing a life threatening illness also raises questions about existence and self-integrity, and 
patients may be overwhelmed as they struggle with loss of control, the meaning of their illness, 
uncertainty regarding the future, and unfulfilled goals.27-30  Cancer radically changes the lives of those it 
afflicts, and the accompanying psychosocial distress can lead to worse health outcomes. 
 
Distress affects patient decision-making. To make decisions about cancer treatment and future care, 
patients must understand the nature of the illness, its prognosis and the available treatment options.  A 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
2 
 
patient’s emotional state, ability to comprehend prognostic data, and how oncologists discuss these data 
all affect the understanding of medical information.  Fearing the loss of hope, patients frequently cope by 
psychologically denying the information’s import.31  Patients may be incapable of processing the 
information they hear,32,33 which in turn decreases message encoding and recall.  Emotion affects 
processing; people who are in negative moods may pay more attention to how messages are given than to 
the content of the messages.34,35  Thus when patients are experiencing high levels of negative emotion, 
and oncologists do not ameliorate this affect, patients may be less likely to understand oncologists’ 
messages.  When patients are emotionally distressed, they cannot process information well and 
decision-making suffers. 
 
Patients feel better when they express emotional concerns and their physicians respond 
empathically.  Disclosure of emotional concerns helps patients in two ways.  First, the expression of 
emotional concerns is therapeutic in itself.  The self-disclosure literature demonstrates a variety of 
positive outcomes associated with expressing personally stressful experiences.36  Particularly after loss or 
trauma, expressions about stressful events result in fewer physician visits and fewer self-reported 
physical ailments.37-41  In women with breast cancer, emotional expression reduced somatic symptoms 
and medical visits.42  Patients assign tremendous symbolic importance to their physician and the quality 
of communication.43  When oncologists respond empathically to patients’ emotions, patients report 
higher satisfaction and better adherence.44  One study used a standardized videotape stimulus to assess 
the effect of physician compassion on viewers' anxiety.  They found that women with breast cancer who 
saw the "enhanced compassion" videotape were significantly less anxious after watching it than women 
in the other group.45  Second, cancer patients with unresolved concerns are more distressed than those 
whose concerns are resolved.46-49  Oncologists can only respond to patients’ concerns that they hear, and 
90% of patients say they want to talk to their doctors about these issues.50,51  Oncologists can promote 
these discussions by asking about patients’ emotional concerns.  In addition, techniques such as active 
listening, using open-ended questions and emotional words, and responding appropriately to patients' 
emotional cues assist in their expression.52  When oncologists encourage their distressed patients to 
express their concerns and then respond appropriately, they help relieve patient suffering and create 
an environment conducive to improved decision-making. 
Effective patient-centered communication improves multiple patient outcomes.  Outside of the 
cancer setting, patient-centered communication has been shown to influence malpractice claims,53 patient 
satisfaction,54 recall of information,55 and clinical outcomes such as diabetic glucose control and 
functional status.56,57  Among cancer patients, quality of communication affects psychological well-
being, with “patient-centered” consultations leading to improved satisfaction and psychological 
adjustment.58  Empathic communication reduces patient anxiety and decreases long-term distress.44,45 
Finally, communication techniques that focus on patients’ emotional states are more likely to elicit 
patients’ concerns about pain or other symptoms, which can then be treated.  All aspects of a patient’s 
care are enhanced with improved communication. 
Effective communication is open, honest and patient-centered. According to Epstein and Street, 
“patient-centeredness” includes four core components: (1) eliciting and understanding the patient’s 
perspective, (2) understanding the patient’s psychosocial context, (3) achieving a shared understanding of 
the problem and its appropriate treatment in the context of the patient’s preferences and values, and (4) 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
3 
 
empowerment, by involving patients actively in decision-making.59,60 This serves as an overall model for 
good communication, and many guidelines have been proposed for specific tasks such as delivering bad 
news, discussing disease progression and talking about palliative care.61-67 68,69 70 Patient-centered 
communication is respectful, empathic, inclusive, and efficient, seeks to elicit patients’ goals and 
preferences, and to match these to an individualized plan of care.  
Oncologist-patient communication often does not meet experts’ standards.  Unfortunately, a large 
literature documents that physicians generally, and oncologists specifically, often do not meet the 
accepted quality standards for communication.43,51,65,71-75  Physicians rarely talk with seriously ill patients 
about their goals, values or even treatment decisions.76-83  Oncologists commonly do not elicit the full 
range of terminally ill cancer patients’ concerns, or attend to patients’ affect.84  Rather than using 
facilitative communication techniques, such as open-ended questions or empathic responses, when 
inquiring about psychosocial issues they often block discussion by changing the subject or not attending 
to patients’ emotions.85  As a result, cancer patients tend to disclose fewer than 50% of their concerns to 
oncologists and other providers.85,86  One study of 177 encounters in which oncologists gave bad news 
revealed that they tended to use closed rather than open-ended questions, and infrequently asked about 
psychosocial concerns.74 In another study, only one of five oncologists' assessments of patients’ distress 
correlated accurately with patients’ self-report.75  A study from the Netherlands of audio-recorded 
encounters between oncologists and patients receiving palliative chemotherapy observed that physicians 
devoted 64% of their conversation to medical/technical issues and only 23% to health-related quality of 
life issues.87  Patients’ emotional functioning and fatigue were not addressed 54% and 48% of the time, 
respectively.  
We conducted the SCOPE Trial (Study of Communication in Oncologist Patient Encounters; R01 
CA100387, Tulsky, PI, Pollak, Arnold, and Farrell, Co-I’s) to analyze emotion handling in patient-
physician communication and to develop an intervention for physicians.88  In the initial observational 
phase, we audio-recorded 398 conversations between 51 oncologists and 270 patients with advanced 
cancer in two cities.  We found that patients disclosed emotional concerns in only 37% of these 
conversations, fewer than expected given the known high prevalence of distress.89 When patients did 
express negative emotions (e.g., anxiety, fear), oncologists responded empathically only 27% of the time.  
Patients likely stop expressing their concerns when they learn they will not receive a helpful response. 
Several oncologist characteristics were associated with their communication style.  Oncologists who used 
more empathic statements were younger than those who did not.  Also, oncologists who responded 
empathically were more likely to describe themselves as socioemotional rather than technical in their 
orientation to medical care (80% vs. 45% respectively, p=.03).  Of note, gender was related to the 
number of empathic opportunities; female patients seen by female oncologists had the most empathic 
opportunities (p=.03), suggesting that oncologist behavior (women oncologists use more empathic 
language) affects the expression of concerns.89  
Oncologists’ may not address patients’ emotional concerns for several reasons.  Constructs in social 
cognitive theory,90 the transtheoretical model of behavior change,91 and a barriers model proposed by 
Cabana92 may explain oncologists’ behavior.  They may be unaware that they are neglecting patients’ 
emotional concerns, lack the skills to address patients’ concerns, feel that addressing their concerns will 
not improve patients’ well-being (outcome expectations), lack the confidence to address patients’ 
emotional concerns (self-efficacy), or be unmotivated to improve their communications skills (readiness 
to change).  Also, external barriers such as patient factors (e.g., patients giving indirect rather than direct 
emotional cues93) and environmental factors (e.g., lack of time or resources) may deter oncologists from 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
4 
 
addressing patients’ emotional concerns.  A significant gap exists between the idealized model of 
oncologist-patient communication and the reality of practice. Behavior change theories explain 
oncologists’ communication patterns, and may be employed in interventions to improve these 
behaviors. 
Training can improve clinician communication.  Although physicians frequently regard the ability to 
communicate as an inborn talent, these skills can be learned.  With one exception,94 rigorous evaluations 
of communication skills teaching have shown positive results when interventions incorporate adult 
learning principles, practice, and feedback in settings supervised by trained facilitators.  The American 
Academy on Physician and Patient (AAPP) first developed the model for such training with a 5-day 
course that included didactic learning, small group skills practice, and self-awareness sessions where 
participants reflected on feelings provoked by clinical situations.  Participants reported a significant 
increase in their interviewing and self-awareness skills.95  Roter and Hall found that physicians 
participating in an eight-hour course used significantly more emotion-handling skills than did untrained 
physicians, and patients of these physicians experienced decreased distress for up to 6 months 
afterwards.96  Examining the effect of course length, Levinson and Roter found that physicians 
completing a 2.5-day course asked more open-ended questions and more frequently asked patients’ 
opinions than physicians enrolled in a half day course.97  Worstell reported that a similarly designed 
interactive seminar for physicians on how to communicate better with regard to asthma led to shorter 
physician-patient encounters and to more favorable patient responses to physicians’ actions.98 
This training model has been used in oncology as well.  Baile conducted a series of half-day workshops 
focused on giving bad news and managing difficult patients99 and found increased confidence in 
communicating bad news and managing problem situation cases after the workshops.  Parle and Maguire 
conducted multi-day workshops in the UK for oncology clinicians that demonstrated improved self-
assessed ratings of performance in communication tasks and improvements in behaviors with 
standardized patients.64  Fallowfield, also in the UK, conducted a randomized controlled trial of a 3-day 
training course involving 160 oncologists from across the country.100,101  She found that, in addition to 
favorably altering oncologists’ attitudes and beliefs towards psychosocial issues, the course resulted in 
improved communication behavior in videotaped patient interviews.101  Intervention oncologists 
increased significantly their use of open-ended questions, expressions of empathy, and appropriate 
responses to patient cues or empathic opportunities. These changes persisted for at least twelve 
months.102 
Members of our team have developed several programs to train physicians to recognize better patients’ 
psychological distress and to respond empathically.  An eight- hour communication skills course for 
medical house staff improved their ability to deliver bad news and respond to emotional cues in 
standardized patient evaluations.103  We have created the leading communication skills training program 
for oncologists, OncoTalk, originally designed as an intensive four-day communication retreat for 
oncology fellows.104  Based on small group learning using role play and feedback, we demonstrated 
significant improvement in fellows’ empathic communication behaviors.105  In fact, changes were so 
profound that the trained fellows were easily identified by blinded raters.  Perhaps most importantly, 
participants found it transformed their practice, and the model has been replicated across the U.S. and 
overseas. Communication skills such as eliciting concerns, handling emotions, and demonstrating 
empathy, can be taught and lead to increased physician self-confidence and reduced patient distress.   

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
5 
 
Face-to-face courses are the gold standard for training physicians, yet have significant limitations.  
Well designed communications skills training workshops are effective in changing physician behavior, 
yet they are costly, and time-intensive.  Such interventions require extensive coordination to assemble 
groups, facilitators and simulated patients.  Although they may be appropriate for physicians in training 
who can be required to attend, it is unrealistic to expect many practicing oncologists to take the time from 
their busy practices to undergo such training.  Practicing clinicians need a more flexible model to fit 
communication skills training into their busy lives.  
The SCOPE program – a widely disseminable online alternative to the face-to-face course.  We 
recognized this need for alternative evidence-based educational venues that are easily accessible, not 
disruptive to clinical practice, inexpensive, and brief.  We developed SCOPE, a computerized, 
interactive, tailored program that teaches oncologists basic communication skills and allows them to 
review their own audio-recorded encounters and provides suggestions on how to respond better to 
patients’ negative emotions.  SCOPE uses the principles found to be effective in face-to-face retreats and 
incorporates them into an online tool that can be efficiently used in the privacy of an oncologists’ home 
or office.106  In SCOPE, oncologists learn basic skills and observe them in exemplar video and then have 
the opportunity to practice and receive feedback on their own conversations.  Furthermore, employing 
the Social Cognitive Theory model described above,90,92 SCOPE addresses oncologists’ self-efficacy and 
outcome expectations, and provides ongoing coaching and encouragement to overcome barriers to 
learning and adopting new skills.   
The original SCOPE program was comprised of five modules: (1) Principles of Effective 
Communication; (2) Recognizing Empathic Opportunities; (3) Responding to Empathic Opportunities; 
(4) Conveying Prognosis; and (5) Answering Difficult Questions.  A final module summarized main 
points from the intervention.  Each module was designed to be viewed in 10-15 minutes and followed a 
similar format.  First, barriers to learning the material were addressed.  Then, the new communication 
skill was introduced, and demonstrated in one or two video clips.  Important teaching points were 
summarized and, finally, users were asked to review selected excerpts from their own previously 
recorded conversations.  These excerpts were accompanied by tailored feedback (e.g., “Great job, try to 
use more statements like that!” or “Next time, you may want to try saying….”).  Oncologists were then 
asked to commit to using a new skill with their patients in the future, and they were sent email reminders 
of this commitment prior to their next clinic sessions.  
In a randomized, controlled trial, we tested SCOPE’s efficacy with 48 oncologists and 264 patients.107  
The program was well accepted. Twenty-one of 24 in the intervention group reported using the CD-ROM 
with a median usage time of 64 minutes. More importantly, 92% of the oncologists reported changing 
their clinical practice as a result of what they learned in SCOPE.106  Oncologists in the intervention arm 
used more empathic statements (RR =1.9; 95% CI, 1.1, 3.3; p=0.02) and were more likely to respond to 
negative emotions empathically (OR = 2.1; 95% CI, 1.1, 4.2; p=0.03) compared to control arm 
oncologist.  Most importantly, patients of intervention oncologists reported higher trust compared with 
patients of control oncologists (estimated mean difference = 0.1; 95% CI, 0.0, 0.2; p=0.04).107  After one 
hour of training with the SCOPE program, oncologists improved their skills and increased their patients’ 
trust, findings almost as robust as seen in multi-day workshops. The SCOPE program, an innovative, 
computerized approach to teaching communication skills based on the same principles as face-to-face 
courses, can achieve similar results to these courses, yet be integrated into the everyday practice of a 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
6 
 
busy oncologist.  SCOPE offers an inexpensive, practical and disseminable alternative to 
communication workshops. 
The Enhanced SCOPE Program – Capturing the patient’s voice in the medical encounter.  In the 
SCOPE program, medical visits between oncologists and patients are recorded, uploaded to a server, and 
using an application called ENCOUNTER, are coded by trained research assistants for the specific 
behaviors of interest (e.g., patient expressions of negative emotion, empathic responses by oncologists, 
etc).  These results are then given back to the oncologists, together with tailored feedback on specific 
examples.   
For this project we have created the Enhanced SCOPE Program.  Rather than having the coding done 
only by research assistants, we will involve patients in the feedback process.  The recorded encounters 
will also be reviewed by trained patient advisors (not the oncologists’ own patients) who will listen to the 
recordings and directly insert their own “subjective” feedback.  Physicians rarely are given the 
opportunity to hear what patients truly think about their communication style.  The Enhanced SCOPE 
Program provides oncologists with both objective coded feedback and a sense of how real patients 
react to their communication, thus “using the patient’s voice” to improve the medical encounter.  
The American Board of Internal Medicine (ABIM) Maintenance of Certification Program – A 
model for integrating Enhanced SCOPE into clinical practice.  All ABIM board-certified medical 
specialists (certified within the past 25 years) are required to enroll and complete the Maintenance 
of Certification (MOC) program requirements if they wish to retain this important credential.  The 
framework and evidence for the MOC program are well-established.108,109  A physician’s 
association with the MOC program occurs over a number of years and involves completing a 
series of steps including educational modules and an exam.  An important and unique component 
of this program is the Practice Improvement Module (PIM), in which recertifying physicians 
conduct a quality improvement exercise using patients and data from their own practices.  There 
are a number of different PIM’s to choose from and, currently, one of these is a communication 
module.  Physicians who select this module ask a sample of their patients to complete the 
Clinicians and Groups-Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) 
satisfaction survey, receive a report of the survey results as feedback, and then design and 
complete a performance improvement plan based on these results.  When they are done with their 
performance improvement plan, CAHPS surveys are collected again from a different sample of 
patients.  The MOC program offers a unique opportunity to integrate the Enhanced SCOPE 
program into a process already familiar to (and required of) board certified internists and 
subspecialists.  As an alternative to the current communication PIM, physicians could audio 
record conversations with their patients using their smart phone (the app is already developed), 
these conversations would be coded and reviewed by the patient advisors, and the physicians 
would then work through the SCOPE modules, including receiving feedback on their 
conversations.  We have engaged in a strong partnership with ABIM around researching this idea 
that allows us to take advantage of the powerful ABIM MOC platform.  
 This collaboration meets the longstanding goal of ABIM to be able to truly evaluate and improve 
communication quality, and the intervention can be widely disseminated that could transform 
patients’ experiences with their physicians. 
 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
7 
 
2.0 AIMS  
The overarching goal of this project is to improve communication between oncologists and their patients 
by ensuring that the patient’s voice is heard in the medical encounter. Thus, we hope to improve the 
experience for patients living with cancer. We seek to accomplish this goal by providing oncologists 
communication skills training that includes feedback on their own audio-recorded conversations. The 
feedback will come from two sources: 1) Professional research assistant coders who will identify 
objective learning opportunities based on specific coding criteria and 2) Trained patient reviewers who 
will listen to the recordings and offer their own, subjective feedback at key moments in the encounters. 
We will address the following specific aims: 
Aim 1: Identify barriers to implementing the ENHANCED SCOPE app. & website in participating 
oncology practices. Barriers include: oncologist enrollment, patient enrollment and transmission of data. 
Aim 2: Compare the effect of the Enhanced SCOPE program to the standard communication Practice 
Improvement Module (PIM) on patient satisfaction with oncologist communication as measured by the 
CG-CAHPS survey.  
Aim 3: Compare the effect of the Enhanced SCOPE program to the standard communication PIM on 
oncologist communication behaviors, including their use of empathic responses to patient emotional 
concerns, as coded from audio-recorded medical encounters. 
 
3.0 RESEARCH SUBJECT SELECTION  
The study will involve the recruitment and consent of three types of subjects: (1) board-certified 
oncologists who practice in any setting in the United States and have a smartphone; (2) patients receiving 
care from these oncologists who are willing to complete an anonymous survey; and, (3) patients 
receiving care from these oncologists who are willing to be audio recorded.   
Inclusion criteria: Eligible patients who will be recorded will have metastatic cancer, (those patients who 
take the survey can have cancer of any stage) be at least 18 years of age, speak and read English, and 
receive oncology care from an enrolled oncologist. Patients who meet criteria will be introduced to the 
study by their enrolled oncologist during a clinic visit. There is no requirement related to life expectancy. 
Eligible caregivers who will be recorded are those present in the room with the enrolled oncologist and 
the patient participant receiving oncology care during the audio-recorded clinic visit. Caregivers will be 
introduced to the study by an enrolled oncologist alongside a recruited patient.  
All participating oncologists who are enrolled in the ABIM MOC program, complete Practice 
Characteristics Survey, complete baseline patient surveys of at least 25 patients and who audio record 4 
patient interactions will be eligible for randomization to study conditions. 
Exclusion criteria: Oncologists who do not speak English to their patients, patients who do not speak 
English, oncologists who do not have enough patients with advanced cancer. DFCI oncologists are 
excluded from this study; this includes DFCI satellite sites, non-DFCI DF/HCC sites, and other non-
DF/HCC sites that are under the DFCI IRB. 
DF/HCC Standard Operating Procedure for Human Subject Research Titled Subject Protocol 
Registration (SOP #: REGIST-101) must be followed. This project has been granted decentralized 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
8 
 
registration where study staff as opposed to the Office of Data Quality staff will be registering 
participants in OnCore. 
 
4.0 RESEARCH SUBJECT ENTRY 
First, we will enroll oncologists through the American Board of Internal Medicine (ABIM) Maintenance 
of Certification (MOC) program. All oncologists involved with the MOC program are eligible for the 
study and will self-refer through the ABIM website, which will house a description of the study and 
invite participation. Additionally, ABIM will contact oncologists enrolled in the MOC program by email 
to announce the offering of the Enhanced SCOPE program. Oncologists interested in participation may 
review detailed information about the study on the website and contact a study representative at Dana 
Farber Cancer Institute or ABIM with questions. Oncologists who express interest will be invited to 
register and assigned an individual sign-in code. They will enter the online Communication Practice 
Improvement Module (PIM) using their code, read and sign an electronic consent form, and complete the 
registration process. Once enrolled, the oncologist will recruit 25 patients from their practice to complete 
anonymous CAHPS satisfaction surveys. Any patient who is seeing or has seen the oncologist at any 
time will qualify as a participant in this study. They can be an existing patient or someone who is seeing 
the doctor for the first time. Because the CAHPS survey is entirely anonymous and no PHI will be 
disclosed, formal written informed consent will not be required for the patients completing the surveys. 
Language at the introduction of the survey will describe the research, be clear that participation is 
voluntary, and state that survey completion implies consent. 
After 25 patients have completed the surveys, oncologists will then identify potentially eligible patients 
from their practice to audio record one clinic visit each (using a secure smartphone application).  Doctors 
will be informed about the importance of privacy while using their smartphones, and will be given 
information explaining how to keep their recordings private and secure. The security measures are as 
follows: 
As a participant in the SCOPE study, the oncologist will audio record up to 10 conversations they have 
with their patients. SCOPE follows strict procedures to ensure the privacy and security of these audio 
recordings, and it is the oncologists’ responsibility to do the same. They will follow these guidelines for 
using the recorder app: 
1. Keep your phone secure: 
- Use only your personal phone, not a shared phone or device. 
- Set your phone’s built-in locking feature to require a passcode, fingerprint, or other identification 
method each time the phone is used. 
- Choose a hard-to-guess passcode and change it periodically and whenever you think it may have been 
compromised. 
- Set your phone to automatically lock after a period of inactivity. 
- Keep your phone up-to-date with the latest operating system. 
2. Keep your recorder app passcode secure: 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
9 
 
- You will be required to set a passcode the first time you use the recorder app, and then enter the 
passcode each time you subsequently use the app. 
- Choose a passcode that is not easily guessed. 
- Do not share your passcode with others or write it down. 
- If you suspect your passcode has been compromised, change it immediately. 
3. Use the recording app securely: 
- Quit or switch out of the app when not in use, it will lock automatically when you do this. 
- Use the app’s lock function when you need to leave the app running unattended. 
- Be aware of the physical environment in which you use the app. If you listen to a recording, make sure 
it cannot be heard by others. 
A total of 4 patient visits (with different patients) will be recorded prior to oncologist randomization and, 
after oncologists have completed the Enhanced SCOPE intervention or the control Communication 
Module, they will audio-record another 4 patient visits (with different patients from the pre-intervention). 
For those oncologists enrolled in the SCOPE intervention, an additional 2 patients will be recorded 
during the intervention period to provide them with additional feedback on their communication as they 
undergo the training. These 2 extra recordings per SCOPE oncologist will be used solely for feedback 
purposes and will not be included in the analysis at the end of the study.  
 
Oncologists will identify potential patients for audio-recorded visits and introduce them to the study, 
however they will not be the ones to obtain informed consent. Rather, the oncologist will download and 
print from the ABIM website a handout with a full written description of the study and give that to the 
patient. This handout will include a phone number connecting the patient to study staff. Oncologists 
will be reminded that their patients may feel obligated to participate because they are being asked by 
their treating physician, and will be asked to reassure the patient that their participation is entirely 
voluntary. If patients wish to proceed, they will then call the DFCI study staff who will introduce the 
study and discuss the basic elements of informed consent with the patient, including the nature of the 
research study, risks, benefits and alternatives to participating. Study staff will describe what PHI may 
be disclosed, to whom it will be disclosed, and the use of the information. Study staff will also make 
clear to the patient that participation is entirely voluntary and may be withdrawn at any time and that the 
decision not to participate will not affect their care, now or in the future. Patients will be encouraged to 
ask questions during the consent process or in the future should they arise later. After expressing their 
willingness to participate, the patients will be sent a link via text or email to an electronic informed 
consent and HIPAA authorization form. Study staff will confirm the receipt of consent, notify the 
oncologist, and after that point, the visit may be audio-recorded. 
If a caregiver is present, the physician may print and download from the ABIM website a handout 
similar to that given to the patient, including a full written description of the study. At the beginning of 
the patient consent process, patients are asked if a caregiver or family member will accompany them 
during the recorded appointment. If a caregiver will be present, they are passed the phone and talked 
through the caregiver consent by study staff. If multiple caregivers are present, they will be talked 
through the caregiver consent at the same time or separately if preferred. All caregivers will have the 
opportunity to ask questions during the consent process or in the future should they arise later. After 
expressing their willingness to participate, the caregiver(s) will be sent a link via text or email to an 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
10 
 
electronic informed consent. All caregivers will utilize the same link and physician ID, but will login to 
the consent form using separate caregiver ID numbers given to them by study staff during the consent 
call. All caregivers present in the room at the time of recording must sign the consent form before the 
visit may be audio-recorded. 
In the case of a patient or caregiver who does not have a phone with text and internet capabilities or is 
struggling to complete the electronic consent process virtually, we will provide to the physician two 
options. A physician can request an iPad with a return envelope addressed to study staff at Dana-Farber. 
This iPad will be pre-programmed with the links to both the patient and caregiver consent. Or, a 
physical copy of the consent form will be included with the handout containing the phone number 
connecting the patient or caregiver to study staff. If they wish to proceed, they will then call study staff 
via the phone number found on the handout. A member of the study staff will introduce the study and 
discuss the basic elements of informed consent with them, including the nature of the research study, 
risks, benefits, and alternatives to participating. In the case of consenting a patient, study staff will 
describe what PHI may be disclosed, to whom it will be disclosed, and the use of the information. Study 
staff will also make clear to the patient that participation is entirely voluntary and may be withdrawn at 
any time and that the decision not to participate will not affect their care, now or in the future. Patients 
and caregivers will be encouraged to ask questions during the consent process or in the future should 
they arise later. After expressing their willingness to participate, study staff will ask the patient or 
caregiver to complete the electronic consent on the iPad provided to the physician or sign and date the 
physical copy of the consent form and hand it back to their physician. The physician will then fax the 
signed consent form to study staff who will in turn sign and date the consent form. Study staff will fax 
the completed consent form back to the physician who will give it to the consented patient or caregiver 
for their records. After that point, the visit may be audio recorded.  
In the case that study staff are present at the physician’s clinic at the time of consenting, study staff will 
talk the patient or caregiver through the consent process in person. They will then hand the patient or 
caregiver an iPad with the appropriate electronic consent form pulled up or a physical copy of the 
consent form and ask them to complete it. Once the consent has been completed, the visit may be audio 
recorded. 
 
5.0 STUDY DESIGN AND METHODS 
 
5.1 Design/Study Type: 
Interventional: Randomized controlled trial 
5.2 Selection of Instruments: 
CG-CAHPS survey: Patient satisfaction with communication – the primary outcome of this trial is the 
effect of the enhanced SCOPE program on patient satisfaction with communication, as measured by the 
CG-CAHPS survey.111,112  This well validated 48-item measure overseen by the Agency for Health 
Research and Quality and endorsed by the National Quality Forum measures patient experience with an 
outpatient health care provider over the past twelve months.  The CG-CAHPS survey is comprised of 
questions that encompass 3 major composites (access to care, doctor communication, and courtesy of 
staff) and two global rating items. All composites have been shown to yield acceptable individual level 
internal consistency reliabilities (Cronbach’s alpha >= 0.77), as well as practice level reliabilities 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
11 
 
(Cronbach’s alpha >=0.75). In addition, all composites were significantly correlated (P<0.01) with the 
global rating items, demonstrating construct validity of the measures. These studies were used in the 
SCOPE study. It should take about 15 minutes for the patients to complete the survey.  
Based on prior work at the American Board of Internal Medicine, the Intercluster Correlation Coefficient 
(ICC) is used as a reliability index for measures collected within physician practices such as patient 
ratings or quality measures of care. The ICC measures the stability of patient ratings based on the 
average ratings of the patient samples within physicians.  This reliability measure increases with sample 
size, as does the precision of the average ratings.  The ABIM employs a very high standard of reliability 
for high-stakes assessments (e.g., certification examinations) of 0.85, so has in the past used a survey 
sample size of 35, which provides reliabilities that range from 0.85 to 0.91 (Figure 1). For the purpose of 
this study and in an effort to recruit somewhat fewer patients, we are willing to accept a slightly lower 
(yet still outstanding) reliability of approximately 0.80. As seen in Figure 1, this can be achieved with a 
sample size of 25, and we will use this for the study. 
Figure 1 
 
 
 
 
 
 
 
 
Oncologist surveys – oncologists will complete brief surveys at baseline (demographics) and after 
completion of the practice improvement module (feedback on the module).  These data will be collected 
directly online through the portal of the Enhanced SCOPE website.  This survey was used in the SCOPE 
study and it should take about 5 minutes to complete. For continuous and ordinal variables (e.g., age, 
years in practice) comparisons will use the Wilcoxon rank-sum test.  For categorical variables (e.g., sex, 
race, practice setting) comparisons will use Chi-square tests. These hypothesis tests are intended for data 
monitoring and quality control. 
The relevant scale used from CAHPS that is treated as continuous is derived the Provider 
Communications composite in our study. The communications composite consists of six items relating to 
frequency of events during interactions between the provider and the patient from the patient’s 
perspective. Each of these items are rated on a 1 to 6 scale (never to always) so the raw score is the sum 
of these ratings. The score range in this case is 6 to 36. The communications items will also be scaled 
using a graded response item response model, the latent variable scores will have a mean of 50 and a 
standard deviation of 10. These scaled scores will be treated as continuous variables with an expected 
range between 20 and 80, with standard errors of approximately ± 2 points.  

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
12 
 
5.3 Description of Intervention  
The primary objective of the Enhanced SCOPE program is to teach oncologists to recognize the role of 
emotion in discussions with cancer patients, to increase their self-efficacy for addressing affective 
concerns, and to provide them with the skills for doing so.  Oncologists are most likely to achieve 
competency in these areas when, in addition to didactic training, they can also observe their own 
conversation and receive feedback on their interactions.  The Enhanced SCOPE program itself is an 
online web application that can be viewed from any computer with internet connectivity.   
We will be conducting a randomized controlled trial to test the impact on patient satisfaction and medical 
visit quality of a communication skills teaching intervention for oncologists that is integrated into the 
American Board of Internal Medicine (ABIM) Maintenance Of Certification (MOC) process. 
Oncologists who choose to enroll in this pilot Practice Improvement Module (PIM) will, as described 
above, complete a baseline questionnaire, send out satisfaction surveys to a sample of their patients, and 
then audio record (using a smartphone application) four clinic visits with four different patients. 
Oncologists who are assigned to the control arm will receive the results of the patient surveys and be 
asked to conduct a quality improvement activity that responds to the survey feedback (the current 
“standard” communication PIM). They will be offered several resources for upgrading their 
communication skills and asked to describe their study plan and the success in its completion. Physicians 
randomized to the control arm will be given the opportunity to conduct the SCOPE training after study 
completion. Oncologists assigned to the intervention arm will receive the survey feedback as well as the 
enhanced SCOPE program which provides an online interactive didactic learning platform and provides 
tailored feedback on their own audio-recorded encounters. The feedback will come from two sources: 1) 
Professional research assistant coders who will identify objective learning opportunities based on specific 
coding criteria (e.g., empathic opportunities, use of open-ended questions) and 2) Trained patient 
reviewers who will listen to the recordings and offer their own, subjective feedback at key moments in 
the encounters. These patient reviewers will be drawn from our stakeholder partners and are active 
patient advocates. They will be treated as members of the research team, paid for the reviews, and are not 
patients of the study physicians. After completing the SCOPE online module, intervention group 
oncologists will audio-record two more clinic visits and receive another round of tailored feedback as a 
“booster.” One month after completing the intervention or control arm processes, oncologists in both 
arms will audio record another four clinic encounters and collect another batch of 25 satisfaction surveys 
from a new sample of patients. Outcomes will be patient satisfaction measures in the survey, and 
communication quality, as measured by objective coding from the recorded encounters.   
For physicians struggling to begin or progress through the study, we will offer individual site visits by 
members of our study staff to encourage and support the physician as they take on this project. During 
these visits study staff will meet with physician office staff to help distribute and collect patient surveys 
and to set up processes to make this easier going forward. They will also consult with the physicians to 
trouble shoot any barriers to audio recording conversations. This opportunity will only be offered to 
physicians who have already enrolled in the study and agreed to participate. We will send an email to all 
participating physicians introducing the new offer and providing contact information for study staff for 
those interested in taking advantage of this opportunity. We will require permission from the physician 
before planning a visit. Participating physicians are not required to accept a visit from study staff.  

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
13 
 
Enrollment: N(Oncologists)= 120, N(Patients)= 900, N(Caregivers)= 900.  In the study we plan to enroll 
up to 120 oncologists, with a goal of achieving a final sample size of at least 100 after attrition. 50 
oncologists will be randomly assigned to each 
study arm. Each oncologist will collect 50 patient 
surveys (25 before and 25 after the intervention) 
for a total of 5,000 surveyed patients. These 
surveys are anonymous and the patients will not 
be required to sign a consent form, therefore they 
will not be included in the overall patient 
enrollment numbers above. In addition, all 
oncologists will audio-record 8 conversations 
(800 enrolled patients) and intervention group 
oncologists will record an additional 2 encounters 
(100 patients) for a total of 900. The majority of 
cancer patients are accompanied by at least one 
caregiver, requiring the addition of a caregiver 
cohort. These caregivers serve a role similar to the 
audio recorded patients in this study, but are not 
the focus of the study and their presence is not 
relevant to the study aims. Prior to releasing all 
the participants for the study, we will release a 
vanguard of 6 participants to conduct pilot testing 
to make sure that all components of the research 
design are working correctly, including data 
collection, audio uploads and the online 
intervention. Barring any complications affecting participation, these pilot participants will be included 
in the formal study, are a part of the sample size calculation, and their data will be a part of the main 
analyses. As they are part of the overall research project and must provide consent, these pilot 
participants are included in the physician enrollment numbers above.  
In addition to testing the feasibility of all the components of the study, this pilot will also provide 
preliminary estimates of the Intracluster Correlation Coefficient (ICC) per measure, response frequencies 
for survey items, standardized mean differences in the patient survey composites, and estimates of group 
rate ratios of the number of empathic responses per number of empathic opportunities. These data will be 
used to re-evaluate our initial values used powering the study and to revise any procedures that will 
improve the quality of the information collected. 
Below, we describe the components and technical processes of the SCOPE intervention.  All technical 
and design work related to app and website development will be performed by People Designs, Inc., the 
Durham based firm led by David Farrell, MPH.  They developed SCOPE, the AVA audio coding 
software, and a number of other applications used in projects with Drs. Tulsky, Pollak, Arnold, and Back. 
Educational Content - SCOPE is comprised of five modules that build sequentially upon one another: 
1) Principles of effective communication; 2) Recognizing empathic opportunities; 3) Responding to 
empathic opportunities; 4) Conveying prognosis; and 5) Responding to difficult questions.  The 
principles and mnemonics taught in these modules are derived from the OncoTalk communication skills 
training program.104,105  Although the SCOPE program identifies oncologists’ opportunities for 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
14 
 
improvement, the overall “tone” of the program is positive and supportive.  It uses the language of 
motivational interviewing, stresses how difficult these conversations are for clinicians, states that 
shortcomings are ubiquitous among physicians, and praises and encourages positive behaviors when 
identified.  Each module includes an introduction to the topic, teaches several concrete skills with video 
demonstrations, provides audio clip examples from the oncologists’ own conversations of when they 
used the skills and opportunities where they could have used the skills if they did not, and offers tailored 
audio feedback encouraging more use of positive behaviors, less use of unhelpful ones, and alternative 
approaches to employ when missed opportunities are detected.  In addition to the objective coded data 
and examples, oncologists will also receive examples with feedback selected by the trained patient 
reviewers. Finally, each module concludes with action steps to which the oncologist commits, and about 
which they will receive an email reminder in the near future.   
Oncologist-Patient Encounter Audio Recording - Oncologists will audio record their patient 
encounters using a smartphone application specifically developed for this purpose.  Upon enrollment, 
providers will receive an email message with links to install the smartphone app. The app is compatible 
with both iOS (iPhone) and Android operating systems. Possession of a compatible smartphone will be a 
criterion for study enrollment. Prior to recording, oncologists will identify potential patients and provide 
them with a flyer prompting them to call study staff at DFCI. Patients will call study staff who will then 
explain the study to patients and provide a website link via text message directing them to an electronic 
consent and HIPAA authorization form that they will then sign. To record an encounter, the physician 
simply launches this easy-to-use app and taps “record.”  At the end of the recording, physicians tap a 
“stop” button and can choose to immediately upload the recording to the “cloud” review platform, or to 
upload it later (e.g., if they don’t have a current internet connection). Delivery status of each recording is 
prominently noted and the physician will be reminded later to upload the recording.  Participating 
oncologists will be given detailed instructions in using the app (although it is quite simple and intuitive), 
reminded that all data transfer is HIPAA compliant, and that study staff will be available by phone and 
email to provide technical support as needed.  
Professional “Manualized” Coding - Study staff will code recorded encounters using the AVA 
software program developed by our team and used previously in over 20 related studies to code over 
6,000 provider-patient encounters.  Using AVA, coding is performed directly on the audio recordings, 
preserving subtleties and context that would be lost through transcript-based coding alone.  Coders 
identify segments within each recording and apply codes to each segment, with related sub-codes being 
applied to each code as appropriate.  For example, a patient utterance such as “I’m so scared” would be 
coded as an “empathic opportunity,” and a subsequent drop-box would allow it to be coded as a direct 
(rather than indirect) example of this.  The coding rules are highly detailed, and our experience from the 
prior SCOPE studies as well as from numerous other project in our shop is that we can train coders to 
achieve high inter-rater reliability with kappa scores greater than 0.8 for most codes.  After the two 
coders are trained, each will code approximately half of the conversations.  In addition, they will both 
code approximately 15% of the encounters to make sure there is no coder drift.  This technique results in 
very rich data that provides a view into specific provider communication at multiple points in each 
encounter.  
Trained Patient Review - Patient reviewers: We believe that the most powerful example of 
engagement in this research is the patient clinic visit audio recording review with their feedback being 
given to the oncologists.  This allows the patient’s voice to be heard directly by the clinician.  These 
reviews will be conducted by our Patient Research Team Members, our Patient Advisory Council 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
15 
 
Members, as well as other patients who we will recruit with the assistance of our remarkable patient 
partners.  These reviewers will evaluate recorded encounters using a smartphone app developed 
specifically for the project. Reviewers will be notified by study staff when an encounter assigned to the 
reviewer is ready for review. After launching the app and logging into the platform, the reviewer will be 
presented with a list of assigned encounters that have not already been completed. Once an encounter is 
opened, an audio playback interface will allow users to play, pause and seek through the recording. At 
any point during playback, the reviewer will be able to pause and insert a comment as a voice memo 
(spoken by the reviewer) or typed message. Upon completion, the reviewer will be able to inspect and 
modify inserted comments and ultimately mark the review as complete. The app will be designed to be 
immediately intuitive and to walk the reviewer through the review process. Reviewers will be trained in 
usage of the app and study staff will be available to provide technical support through email and phone.  
Each conversation will be reviewed by one patient reviewer.  Prior to conducting any reviews, Dr. 
Pollak, a national expert and trainer in motivational interviewing, will train all patient reviewers.  The 
purpose of this training will be to help reviewers understand how to frame their comments in behavioral 
language that is likely to be better understood and internalized by the oncologists (e.g., “I liked the way 
you said, ‘That must be tough’ – it showed that you really heard what the patient was saying” rather than, 
“You sounded really caring there.”).  While the patient perspective is essential and we need to retain their 
“voice” in this feedback, we are also aware that diffuse or overwhelmingly negative feedback is likely to 
trigger resistance, thus making the patient reviews counterproductive.  We are certain that we can strike 
the proper balance with the very high quality patient partners we have recruited to this study. 
Coders and patient reviewers will be contracted by DFCI. They will receive privacy and security training 
and will follow all DFCI policies and procedures related to accessing study data and maintaining privacy 
and security. Recordings and other data will not be transferred to the coders and patient reviewers. They 
will access recordings only through the AVA web-based coding program, using an encrypted SSL 
connection. Their access to data will be limited to the recordings themselves and non-personally-
identifying ID's used to refer to the recordings. All professional staff coders and patient evaluators will 
be HSP certified.  Patient evaluators will be logged into a database by DFCI staff.  Staff will ensure that 
each evaluator has completed the HSP training. All training certificates will be stored in the Regulatory 
Binder. 
Coded Data Review and Recorded Example Selection - Study staff will be notified when coding (by 
professional coders and patients) has been completed for each encounter.  Data from manualized coding 
will be inspected by Dr. Pollak who may reject, accept, or modify specific coding.  Based on prior 
experience managing similar review processes, initial encounters will require regular checking and 
discussion with reviewers and coders. This will serve as a quality improvement step that leads to better 
review and coding over time, ultimately resulting in very little need for checking these data prior to their 
usage in the online training.  There will be no more than two examples given for each skill taught.  
Similarly, staff will examine patient reviewer comments and where necessary clarify these through 
discussion with the reviewer.  Specific comments may be edited to better reflect reviewer intentions, and 
voice memos may be converted to text when a voice memo needs to be adjusted.  Up to three patient 
reviewer comments per encounter will be selected for presentation to the oncologist. We have found in 
prior work that too much feedback can be overwhelming and physicians are most likely to act on small 
amounts of focused feedback.   
Online Provider Training – After all of an oncologist’s recorded encounters have been collected and 
coded, the data will be transferred to an individually tailored web/mobile-based training program.  A 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
16 
 
SCOPE program account will be created for the oncologist who will be notified via email. The program 
will include all of the components described in the “Educational Content section”, including individual 
feedback on providers' own recorded encounters.  For the manualized coding, each coded oncologist 
behavior will match with a feedback response from a sample library of choices.  For example, when an 
oncologist misses an empathic opportunity, we will provide a sample empathic response.  Oncologists 
will receive patient reviewer comments either in writing or verbally, along with clips from the encounters 
to which those comments are directed. Feedback will reward providers for demonstrated skills and 
identify deficiencies with concrete suggestions to improve communication in those areas.  
The training program will be designed for use on multiple platforms, including computers, tablets, and 
smartphones. The interface will be intuitive, with emphasis on quick digestion of content. Physicians can 
work through modules in a linear fashion or follow their own preferred trajectory using a menu system. 
They will be encouraged to visit all content, and completion will be clearly indicated through a progress 
meter and map showing completed and incomplete sections. Past experience suggests most users will 
complete the intervention in a single setting, but users will have the opportunity to complete it over 
multiple sessions and to return to already completed content at any time.  The training program will track 
usage at the page/screen level, allowing for detailed analysis of each provider's dose and usage trajectory. 
An administration panel will allow research staff to track physician usage. 
Study Feasibility: We believe this program is feasible because of the resources we have to conduct the 
trial. Both interventions the SCOPE program and the Subspecialty Communications Practice 
Improvement Module (PIM) have been field-tested. SCOPE was evaluated in a previous randomized trial 
(107) and the Subspecialty Communications has been operational in the Maintenance of Certification 
(MOC) program at the ABIM for more than 10 years. We have an adequate pool of oncologists and their 
patients to participate in the study. As of 19 April 2016, there are over 6,600 practicing oncologists 
enrolled in the MOC program. ABIM made arrangements with its platform service provider to dedicate 
sufficient computer storage and data collection via web-based and IVR systems to accommodate 150 
physician participants and at least 10,500 patient surveys. This will be more than adequate to cover the 
trial of 120 oncology practices. Finally, the incentives of 40 MOC points out of the required 100 points 
for to maintain certification, 40 CME credits, and a small payment of $75 per physician should enhance 
our ability to accrue physician participants for the study.  
5.4 Data Collection  
Four types of data will be collected in this study: oncologist surveys, patient surveys, oncology visit 
audio recordings, and Enhanced SCOPE program usage data 
 
Oncologist surveys – oncologists will complete brief surveys at baseline (demographics) and after 
completion of the practice improvement module (feedback on the module). These data will be collected 
directly online through the Enhanced SCOPE website, described below and will be stored on the SCOPE 
server. All oncologists, whether assigned to the intervention or control arm, will navigate the study via 
this site. Survey data do not contain PHI or other patient data. These data will be transferred securely to 
ABIM for analysis together with the other data.  

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
17 
 
Patient surveys – the only survey data collected from patients is from the anonymous CAHPS surveys 
administered through ABIM. Patients will be identified by their oncologists and given a copy of the 
survey that on the cover sheet offers two different options through which to complete the survey. One 
option is automated interactive voice response (IVR), in which the patient calls a toll-free telephone 
number and enters an identification number for their oncologist, and is directed through a survey by a 
computerized system. The second option is to complete the survey via an anonymous online portal, for 
which the patient is given the URL and the oncologist identification number. These data, too, will reside 
only at ABIM. The surveys are linked to the oncologist, but otherwise have no identifiable data or 
protected health information. We are requesting a waiver of written consent to collect the anonymous 
survey data via CG-CAHPS from patients who receive care from participating oncologists. We believe 
this is the best way to protect patient confidentiality, as the only linkage between the survey data and the 
patients would be the informed consent document. This way, the data remains truly anonymous. Patients 
will not have the option of completing surveys on paper, thereby eliminating opportunity for loss or 
disclosure of hard-copy data. 
Oncology visit audio recordings – the clinic visit recordings will be collected via the AVA application as 
described below and will reside on the AVA servers. Through review and coding of these recordings 
using the AVA application, additional “code” data will be generated and will be stored on the same AVA 
servers. These code data do not contain PHI or PII and are stored separately from the audio recordings. 
The code data includes data collected from both the patient and from the caregiver(s) during the audio 
recorded clinic visit. The data extracted from the recordings is dependent on the oncologists, their 
responses, and their communication. The coded data are not differentiated into “patient” and “caregiver” 
data; all codes are categorized only by oncologist. Therefore, all data collected from the patient or from 
the caregiver(s) during one audio recorded visit will be considered one set of data. The code data 
(without recordings), will be transferred securely to ABIM for analysis together with the other data. 
Portions of the visit recordings will be available for oncologists to review as part of their usage of the 
Enhanced SCOPE website, with each oncologists having access only to their recordings (the recordings 
of their own clinical encounters). These partial recordings will be stored on the SCOPE server. 
 
SCOPE program usage – the Enhanced SCOPE website will track oncologist usage, generating common 
usage data including login times and content viewed, and will be stored on the SCOPE server.  Usage 
data do not contain PHI or other patient data. Usage data, will be transferred securely to ABIM for 
analysis together with the other data.  
 
 Long-term storage of Data 
Data stored on the SCOPE server and AVA servers will reside there only for the periods they are 
required to be there for study usage. Data will be securely removed from these servers on a per-item basis 
(as users complete the study their data will be removed, rather than waiting for all users to complete the 
study). Removed data will be securely transferred to DFCI servers for long-term storage.   
 
The Enhanced SCOPE Website 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
18 
 
The Enhanced Scope website is custom website developed for the study. The website and associated data 
will be hosted on a HIPAA-compliant server managed by Armor (armor.com), a HITRUST-certified 
HIPAA hosting and data management provider. 
The AVA Application 
The AVA application is a commercial product that will be licensed by the study for use to collect and 
code audio recordings. AVA includes a SaaS web application that is used to manage and code audio 
recordings, generate reports, explore recordings and data, and compile data for further analysis. AVA 
also includes an iOS recording app that transfers recordings directly to the AVA servers (instead of using 
a general purpose audio recorder app). AVA and associated data are hosted on HIPAA-compliant servers 
managed by Armor (armor.com), a HITRUST-certified HIPAA hosting and data management provider.  
 
5.5 Description of Study Process 
5.51 Instrument Administration:  
All instruments in this intervention will be self-administered. [PLEASE SEE SECTION 5.4 “DATA 
COLLECTION” for further details about the surveys and recordings.]  
Once oncologists have enrolled, they have 1-2 weeks to complete a baseline questionnaire, request 
satisfaction surveys from a “convenience sample” of 25 of their patients, and then audio record (using a 
smartphone application) four clinic visits with four different patients.  Within 2-4 weeks the oncologists 
will receive their survey feedback and will be randomized into control and intervention arms. Those 
doctors who are randomized into the control group will have 6 weeks to conduct a quality improvement 
activity that responds to the feedback (the current “standard” communication PIM). Oncologists assigned 
to the intervention arm will have 2 weeks to complete the enhanced SCOPE program that provides 
feedback on their audio-recorded encounters via a web based interactive program. Oncologists in the 
SCOPE arm will record an additional two clinic visits to receive a “booster” of feedback after the online 
training. One month after reviewing their feedback, oncologists in both arms will audio record another 4 
encounters and request satisfaction surveys from a new sample of patients.   
5.52 Intervention Administration:  
The ABIM provides electronic information to potential oncologists and those doctors self-select to join 
the project. Oncologists enroll, consent online, choose their own patients, then proceed with the 
intervention at their own pace with the use of the online function and smartphone applications. Doctors 
continue with this process until they finish their section, the intervention lasts the length of the study.  All 
technical and design work related to smart phone application and website development will be performed 
by People Designs, Inc., the Durham based firm led by David Farrell, MPH. Data from manualized 
coding will be inspected by Drs. Tulsky and Pollak and these doctors will also train patient reviewers to 
code the audio recordings. 
5.53 Special Concerns 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
19 
 
The risks in this study are minimal and non-medical in nature. The primary risk is loss of confidentiality. 
To minimize the likelihood of a breach, we will collect only electronic data and anonymous data when 
possible. Also, mobile recording devices/applications will be designed with security measures in mind. 
Only the minimum amount of PHI necessary will be collected from study subjects, including oncologists 
and patients, and all data will be transmitted via secure, institutionally approved methods. 
5.54 Compensation 
Physicians will be incentivized to participate by receiving 40 MOC points (twice what is usually awarded 
for completing a Practice Improvement Module), 40 CME credits, and a $75 credit toward their MOC 
fee.  
5.6 Adverse Reactions and Their Management 
5.61 Reporting Adverse or Unanticipated Events 
The PI will monitor and report Adverse Events (AEs) to the Dana Farber Office of Sponsored Research.  
5.62 Anticipated Reactions 
This study presents minimal risk to participants. 
5.63 Reaction Management 
Ongoing mechanisms for monitoring the occurrence of AEs include: (1) day-today oversight of study 
activities by the project manager; (2) a toll-free number participants may use to contact study personnel 
at Dana Farber; and regular team meetings and check-ins with project staff and investigators. 
Additionally, all research staff will be Human Subjects certified.    
 
6.0 STATISTICAL ANALYSIS 
6.1 Primary and secondary Aims:  
The overarching goal of this project is to improve communication between oncologists and their patients, 
and thus improve the experience for patients living with cancer. We hope to accomplish this goal by 
providing oncologists communication skills training that includes feedback on their own audio-recorded 
conversations. We hope to accomplish the following specific aims: 
 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
20 
 
Aim 1: Identify barriers to implementing the ENHANCED SCOPE app. & website in participating 
oncology practices. Barriers include: oncologist enrollment, patient enrollment and transmission of data. 
 
Aim2: Compare the effect of the Enhanced SCOPE program to the standard Communication Practice 
Improvement Module (PIM) on patient satisfaction with their oncologist’s communication as measured 
by the CG -CAHPS. 
 
Aim 3: Compare the effect of the Enhanced SCOPE program to the standard PIM on oncologist 
communication behaviors, including their use of empathic responses to patients’ emotional concerns, as 
coded from audio- recorded medical encounters.   
 
 6.2 Sample size and statistical power or precision associated with the sample size 
We considered several outcomes for power calculations to determine study sample sizes, number of 
physicians and number of patients within physician practices. All power calculations were completed 
using the PASS© power analysis 116The study assumptions for the power calculations follow: 
• 
Intervention: SCOPE (Studying Communication in Oncologist–Patient Encounters) program 
• 
Active control: Subspecialty Communications Practice Improvement Module program 
• 
Type I error rate ≤  = .05, two-tailed test 
• 
Power to detect minimally important difference in means, rates or odds ratios ≥ (1-β) = .8 
• 
Analyses will involve random effects models comparing intervention and control groups with 
respect to mean differences, the odds ratios, or rate ratios. 
• 
Sample allocations between treatment groups will be 1:1 
• 
Mean differences in continuous variables (e.g. mean patient satisfaction with communication 
score) will be assessed using a random effects, ANCOVA with covariates 117, 118, 119Minimally 
important differences between groups (in standardized units SD) reviewed included: .20 SD and 
.25 SD. The assumed values for minimally important differences for continuous variables is based 
on effect sizes determined from previous research of the Duke investigators 107and ABIM 
investigators  120 
We planned on meaningful effect sizes to be somewhere between .21 and .25 typical of effect 
sizes we have observed in ABIM studies and representative of the expected effect sizes we 
observe from continuing medical education programs. In a systematic review of the effect sizes of 
CME program a  2009 Cochrane Collaboration review of 30 randomized trials investigating the 
effects of CME on professional practice and healthcare outcomes (versus no CME), the median 
risk difference (RD) was 6% (IQR: 1.8% to 15.9%). Assuming equal sample sizes and a baseline 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
21 
 
risk of 70%, a standardized difference effect size equivalent to a 6% RD is approximately .14 (0.4 
to .33) SD. 
• 
Odds ratios for categorical or ordinal variables (e.g. groups differences between items or rating 
scales from CAHPS survey) will be assessed using a random effects logistic regression 121These 
minimally important effect sizes between groups reviewed for power calculations included 2.0 
and 2.5 and were based in previous research by Duke investigators 107Based on estimates from 
previous Communications PIM performance at the ABIM, we assumed Physician performance 
from the PIM would be about .7 for binomial variables.    
• 
Between group rate ratios of the number of empathic responses per number of empathic 
opportunities will be assessed using a random effects Poisson regression 122Minimally important 
rate ratios based on previous research of Duke investigators 107reviewed included: 2.0 and 2.5. 
Assumed the Poisson regression fits the data so the over-dispersion parameter for the regressions 
was fixed at 1.0 with a baseline empathic response rate of .4 per every one oncologist to patient 
conversation. 
• 
The data will have clustered structure with patient responses or physician conversations nested 
within oncologists. The expected intra-cluster correlation (ICC) is .1. The assumed value of the 
ICC is based on the observed average intra-cluster correlations from patients samples of CG-
CAHPS surveys of physicians completing the Subspecialty Communications Practice 
Improvement Module program at the ABIM and is consistent with ICC values reported nationally 
for the CG-CAHPS 3. 127. 
• 
Cluster sizes reviewed include 4 for analyses of outcomes from physician audio recording 
Possion and 25 per physician for analysis of outcomes from patient responses to CG-CAHPS 
composites.   
• 
It is expected that there we be no more than a 16% loss to follow-up among participating 
physicians over the duration of the study so an initial sample of 120 would ensure 100 oncologists 
remaining for the duration of the study. The most likely reasons for dropouts expected include 
physicians not completing the data collection requirements before randomization or failing to 
complete in the communication improvement activities of the study arms.    
Based on these assumptions, we determined that 100 physicians with 4 patient audio recordings at 
baseline (before randomization) and post-intervention and 25 patient responses to the CG-CAHPS 
surveys at baseline and post-intervention were suitable for our study objectives. Table 1 shows 
the twelve study scenarios that we determined sample size requirements. One hundred physicians 
(50 per study arm) met or exceeded sample size requirements each of the study outcomes 
reviewed. The proposed study size: 100 physicians, 8-10 physician-patient audio recordings 
depending on the arm in which they are enrolled and 50 patient surveys seemed manageable for a 
proposed 18 months of data collection.  
Table 1. Physicians and Patients Required to Detect Selected Effect Sizes with at least 80% 
Power at  = .05 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
22 
 
 
 
6.3 Stratification factors and intervention allocation plan for randomized studies:  
All participating oncologists who are enrolled in the ABIM MOC program, complete Practice 
Characteristics Survey, complete baseline patient surveys of at least 25 patients and who audio record 4 
patient interactions will be eligible for randomization to study conditions. The randomization of 
physicians will occur at the ABIM. The responsible statistician at ABIM will use a computer program for 
randomization. The randomization process will involve stratifying participants by gender and time 
interval blocks (about 12/month) as they enter the project. Because of the administration time required to 
score the baseline audio recordings, physicians will be assigned in time blocks to accommodate the flow 
of work. Total expected time to accrue physician participants is about 16-18 months. 
6.4 Stratification factors and their impact on design:  
Per Group 
Total 
Model 
Effect 
Size 
Rho 
Cluster 
Size 
,     
2-tail 
1 -  
# 
Physic
ians 
# 
Patients 
# 
Physic
ians 
# Patients 
1 
Mean 
Difference 
0.21 SD 
(1.0) 
0.1 
25 
.05 
0.80 
50* 
2,500 
100 
5,000 
2 
Mean 
Difference 
0.250 SD 
(1.0) 
0.1 
25 
.05 
0.81 
36* 
1,800 
100 
3,600 
3 
Poisson 
Reg. 
Rate (2.0) 
0.1 
4 
.05 
0.8 
25 
100 
50 
200* 
4 
Poisson 
Reg. 
Rate (2.5) 
0.1 
4 
.05 
0.8 
14 
56 
28 
112* 
5 
Logistic 
Reg. 
OR (2.0) 
0.1 
4 
.05 
0.72 
50 
200 
100 
400* 
6 
Logistic 
Reg. 
OR (2.5) 
0.1 
4 
.05 
0.8 
37 
148 
74 
296* 
7 
Logistic 
Reg. 
OR (2.0) 
0.1 
25 
.05 
0.8 
26 
650 
52 
1300* 
8 
Logistic 
Reg. 
OR (2.5) 
0.1 
25 
.05 
0.8 
16 
400 
32 
800* 
 *Bolded number determined from power calculations 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
23 
 
At Baseline, participating oncologists will complete a background questionnaire covering the flowing 
topics:  
• 
Oncologist demographics:  At baseline, oncologists will be asked their age, gender, race, 
ethnicity and religion. 
• 
Oncologist experience:  At baseline, oncologists will be asked the number of years they 
have been in practice and the number of hours they spend in direct patient care. 
• 
Oncologist communication training:  At baseline, physicians will be asked about prior 
communication skills training they have received in medical school, residency, fellowship and 
practice. 
• 
Oncologist socioemotional orientation: At baseline, physicians will be asked 2 items that 
assess whether they are more inclined toward the social and emotional aspects of patient care or 
more inclined toward the technological and scientific aspects.129  In the SCOPE Trial, these were 
found to be strong predictors of physician likelihood to use empathic responses. Because 
participating oncologists are enrolled in the ABIM MOC program, administrative data from 
ABIM records will be used to supplement information from the questionnaire particularly training 
and certification performance. ABIM records may also serve to validate responses to the study 
questionnaire or serve as imputation sources in the event of incomplete responses. 
The questionnaire data will be used to assess whether certain training, demographic, or personal 
characteristics serve as moderating variables to the treatment effects. Many to these variables will serve 
as covariates in the ANCOVA analyses or if there are complex interactions observed, the analyses may 
include blocking respondents using one or more of these variables interactions.    
6.5 Early stopping rules, if appropriate:  
NA, there are no plans for early stopping of the study. 
6.6 Definition of and allowance in design for unevaluable/ineligible participants:  
Unevaluable participants will include those physicians who do not complete the baseline data collection 
requirements before randomization. These individuals will not be allowed to participate in the trial. 
Individuals who do not complete both the baseline and re-measurement portions of the study are not 
eligible to receive the 40 MOC points towards their certification status nor will they receive the $75 
monetary incentive to participate or the 40 CME credits. Records of data collected from physicians who 
completed all baseline requirements and were randomized to a study arm but failed to complete the re-
measurement portion will still be used in the analyses as incomplete records. Missing data models will be 
used to adjust for these incomplete records.          
6.7 Analysis plan:  
The study design for analyses purposes is what Murray 124 refers to as a cross-sectional, nested Pretest-
Posttest Control Group Design. The analyses of the primary outcomes, average, patient communication 
and trust scores will be achieved using a linear mixed model with patient responses nested within 
physicians. The test for the main intervention of SCOPE versus Communication PIM is evaluated by 
comparing the variation among the time by intervention against the variation of the time by physician 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
24 
 
interaction. Since there are only two times and two interventions the test is reduced to a t-test comparing 
the differences between post-intervention mean and baseline mean for the score intervention to the same 
difference for the Communication PIM. The advantage to this design when assumptions are met is that it 
allows for adjustment of any pretest and post-intervention differences between groups reducing sources 
of bias that may exist. The cost is some loss in power so we chose to use a larger physician sample of at 
least 100. Table 2 below shows the expected mean squares table for an unadjusted nested, cross-
sectional, Pretest-Posttest Control Group design used in the study. The only difference in the expected 
means squares shown in Table 2 and those from an adjusted (baseline or post-intervention) analyses is 
the reduction in the degrees freedom from the either the group level covariates or patient level covariates 
along with the effects of the covariates removed from the mean squares. 
The analyses of the secondary outcome, oncologists’ empathic response to patient and caregiver 
expressions of negative emotions, a rate variable, proceeds in the same manner as the primary outcome. 
The difference between the two analyses is that the general linear model for the average score outcome 
uses a normal error linear model while the linear model for the secondary rate function follows an 
Poisson distribution with a log link function, thus the error terms for the two analyses have different 
distributions but the same data analyses design.  
The differences we are measuring are the treatment arm differences in physician post-test scores 
controlling for pre-test scores that is, the differences between post-intervention mean and baseline mean 
for the SCOPE intervention to the same difference for the Communication PIM while partialling out the 
effects of relevant covariates. We compare test for effect 1 (Intervention, shown in Table 2) to that for 
test of 4 (time by intervention shown in Table 2) 
Table 2. Expected Mean Squares for an Unadjusted Analysis of Data from SCOPE Versus 
Communication PIM Study 
Source of Variation 
Df 
E(MS) 
MS 
Intervention (Scope vs. PIM):  I 
2-1 
2e + mt2P:i  + 
mtp2e 
MSI 
Physicians: P  
2*(50-1) 
2e + mt2P:I 
MSP:I 
Time, 1=baseline, 2= post-intervention: T 
2-1 
2e + m2TP:I  + 
mPI2T 
MST 
T*I 
1*1 
2e + m2TP:i  + 
MP2TI 
MSTI 
T*P:I 
1*2*(50-1) 
2e + m2TP:I 
MSTP:I 
Patient Observation: T*P:I (8 obs./physician): m  
or 
2*50*2*(4-1) 
2e  
MSe 
Patient Observation: T*P:I (35 obs./physician): 
m 
2*50*2*(25-1) 
2e 
MSe 
Note: df= degrees of freedom, E(MS) = expected mean square, MS = mean square  
 
There are several assumptions that apply to the models discussed 124.  First, variables adjusted in the 
analyses are assumed to have a straight-line relationship with the outcome measures. This assumption 
may be assessed by visual inspection of scatter plots of the covariates against the outcome measures. If 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
25 
 
nonlinear trends are suspected then adding nonlinear terms to the analyses models and checking for 
significance of the coefficients for the terms may be used. Another assumption of the model is that the 
error terms follow either a normal distribution for the primary outcome or a Poisson distribution for the 
secondary outcomes. Model residuals can be examine using quantile to quantile plots comparing the 
actual residuals to expected residuals from the appropriate standard distributions. Third, regression slopes 
for the covariates are assumed to be homogenous across the study conditions. This may be assessed by 
introducing covariate by outcome interaction terms into the models and testing the significance of the 
terms. Fourth, covariates are assumed to be homogeneous across physician groups within conditions. 
This too may be assessed by testing for the significance of a physician by covariate interaction. Finally, 
the models assume that the outcomes have no effect on the covariates. There is no good way to assess 
this possibility. We must be careful to pick covariates that are related to the outcome but are not likely to 
be affected by the interventions such as demographic characteristics like gender. Finally, because many 
the assumptions of these models are assessed by adding additional terms, the model fit statistics such as 
the likelihood ratio test or the Akaike’s and Schwarz’s criteria statistics. The fit statistics of the various 
models may be plotted as a function the number of model parameters to suggest whether adding these 
terms for assumption testing is reasonable. All analyses will be conducted using the SAS® statistical 
software, version 9.4 
6.8 Handling of missing data in the analysis:  
Missing data either incomplete responses (items missing) or absent responses (dropouts or missing 
records) reduce the efficiency of the studies and may introduce biases into the analyses. Little and Rubin 
125 describe a hierarchy of missing data mechanisms including missing completely at random (MCAR), 
missing at random (MAR), and not missing at random (NMAR). When data are MCAR then the records 
even those with missing values can be considered a random sample of the complete data 126. Thus, a 
complete case analyses of only observations that all data elements would give you accurate estimates of 
the target population. General linear models and generalized linear models are valid assessment tools in 
the MCAR case 126, 127. With MAR data, complete case analyses is invalid however, missing data may be 
imputed from the observed data even from records with incomplete responses. Valid inference for MAR 
can only occur in generalized linear models or generalized mixed linear models if the models are 
correctly specified otherwise the inferences will be biased 126, 127.  When data are NMAR, then the 
mechanism for missing data must be included in the model but unfortunately the missing mechanism is 
usually not known so and would have to be simulated 126. There is no way to correctly analyze NMAR 
data unless the missing mechanism is included the analytical model.  
For our purposes we will assume that missing data are MAR and use multiple imputation methods such 
as Markov chain Monte Carlo method which creates a series of plausible values for the missing data 127. 
The data are combined and estimates for inference are made. The benefit of imputation is that a set or 
reasonable values are substituted for the missing one. The cost of imputation is that the error terms for 
estimates such as differences between interventions are increased thereby reducing the power of the 
statistical tests. For sensitivity analyses, we plan to compare results from three basic analyses: 1) 
complete case analyses (assumes MCAR is true), 2) multiple imputation by MCMC of all missing values, 
and 3) multiple imputation by MCMC to create a monotonic missing patterns in the data so that different 
imputation models may be tried to see how study results vary under different imputation models 125. If 
after close inspection of the data, the research team suspects that a NMAR mechanism may be at work, 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
26 
 
we will test the hypothesized NMAR assumption through simulation to evaluate how results might 
change.  
6.9 Multiple Testing for Subgroup and Exploratory Analysis 
Hypothesis tests of the differences between study interventions primary and secondary outcomes will be 
tested using a two, tailed significance test with the Type I error rate set at .05. For subgroup and 
exploratory analysis adjustments for multiple testing should be made. We intend to adjust for multiple 
testing by controlling the proportion of hypotheses falsely rejected among all rejected hypotheses, the 
false discovery rate, using Benjamini and Hochberg’s linear step-up method 128. 
 
 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
27 
 
7.0 REFERENCES:  
 
1. 
Carlson LE, Groff SL, Maciejewski O, Bultz BD. Screening for distress in lung and breast cancer 
outpatients: a randomized controlled trial. J Clin Oncol 2010;28:4884-91. 
2. 
Zabora J, BrintzenhofeSzoc K, Curbow B, Hooker C, Piantadosi S. The prevalence of psychological 
distress by cancer site. Psychooncology 2001;10:19-28. 
3. 
Gao W, Bennett MI, Stark D, Murray S, Higginson IJ. Psychological distress in cancer from 
survivorship to end of life care: prevalence, associated factors and clinical implications. Eur J 
Cancer 2010;46:2036-44. 
4. 
Wilson KG, Chochinov HM, McPherson CJ, et al. Suffering with advanced cancer. J Clin Oncol 
2007;25:1691-7. 
5. 
Cleeland CS, Gonin R, Hatfield AK, et al. Pain and its treatment in outpatients with metastatic 
cancer. N Engl J Med 1994;330:592-6. 
6. 
Coyle N, Adelhardt J, Foley KM, Portenoy RK. Character of terminal illness in the advanced cancer 
patient: pain and other symptoms during the last four weeks of life. J Pain Symptom Manage 
1990;5:83-93. 
7. 
Weiss SC, Emanuel LL, Fairclough DL, Emanuel EJ. Understanding the experience of pain in 
terminally ill patients. Lancet 2001;357:1311-5. 
8. 
Derogatis LR, Morrow GR, Fetting J, et al. The prevalence of psychiatric disorders among cancer 
patients. JAMA 1983;249:751-7. 
9. 
Fallowfield L, Ratcliffe D, Jenkins V, Saul J. Psychiatric morbidity and its recognition by doctors in 
patients with cancer. Br J Cancer 2001;84:1011-5. 
10. Grabsch B, Clarke DM, Love A, et al. Psychological morbidity and quality of life in women with 
advanced breast cancer: a cross-sectional survey. Palliat Support Care 2006;4:47-56. 
11. Kissane DW, Clarke DM, Ikin J, et al. Psychological morbidity and quality of life in Australian 
women with early-stage breast cancer: a cross-sectional survey. The Medical journal of Australia 
1998;169:192-6. 
12. Spiegel D. Cancer and depression. The British journal of psychiatry 1996:109-16. 
13. Stark D, Kiely M, Smith A, Velikova G, House A, Selby P. Anxiety disorders in cancer patients: 
their nature, associations, and relation to quality of life. J Clin Oncol 2002;20:3137-48. 
14. Badger TA, Braden CJ, Mishel MH. Depression burden, self-help interventions, and side effect 
experience in women receiving treatment for breast cancer. Oncol Nurs Forum 2001;28:567-74. 
15. Badger TA, Braden CJ, Mishel MH, Longman A. Depression burden, psychological adjustment, and 
quality of life in women with breast cancer: patterns over time. Research in nursing & health 
2004;27:19-28. 
16. Wells KB, Stewart A, Hays RD, et al. The functioning and well-being of depressed patients. Results 
from the Medical Outcomes Study. JAMA 1989;262:914-9. 
17. Breitbart W, Rosenfeld B, Pessin H, et al. Depression, hopelessness, and desire for hastened death in 
terminally ill patients with cancer. JAMA 2000;284:2907-11. 
18. Kurtz ME, Kurtz JC, Given CW, Given B. Relationship of caregiver reactions and depression to 
cancer patients' symptoms, functional states and depression--a longitudinal view. Soc Sci Med 
1995;40:837-46. 
19. Simon GE, VonKorff M, Barlow W. Health care costs of primary care patients with recognized 
depression. Archives of general psychiatry 1995;52:850-6. 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
28 
 
20. Valdimarsdottir U, Helgason AR, Furst CJ, Adolfsson J, Steineck G. The unrecognised cost of 
cancer patients' unrelieved symptoms:a nationwide follow-up of their surviving partners. Br J 
Cancer 2002;86:1540-5. 
21. DiMatteo MR, Lepper HS, Croghan TW. Depression is a risk factor for noncompliance with medical 
treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch Intern 
Med 2000;160:2101-7. 
22. Fulop G, Strain JJ, Vita J, Lyons JS, Hammer JS. Impact of psychiatric comorbidity on length of 
hospital stay for medical/surgical patients: a preliminary report. The American journal of psychiatry 
1987;144:878-82. 
23. Levenson JL, Hamer RM, Rossiter LF. Relation of psychopathology in general medical inpatients to 
use and cost of services. The American journal of psychiatry 1990;147:1498-503. 
24. Fawzy FI, Fawzy NW, Hyun CS, et al. Malignant melanoma. Effects of an early structured 
psychiatric intervention, coping, and affective state on recurrence and survival 6 years later. 
Archives of general psychiatry 1993;50:681-9. 
25. Loberiza FR, Jr., Rizzo JD, Bredeson CN, et al. Association of depressive syndrome and early 
deaths among patients after stem-cell transplantation for malignant diseases. J Clin Oncol 
2002;20:2118-26. 
26. Spiegel D, Bloom JR, Kraemer HC, Gottheil E. Effect of psychosocial treatment on survival of 
patients with metastatic breast cancer. Lancet 1989;2:888-91. 
27. Blanchard CG, Ruckdeschel JC. Psychosocial aspects of cancer in adults: implications for teaching 
medical students. J Cancer Educ 1986;1:237-48. 
28. Cassell EJ. Diagnosing suffering: a perspective. Ann Intern Med 1999;131:531-4. 
29. Cherny NI, Coyle N, Foley KM. Suffering in the advanced cancer patient: a definition and 
taxonomy. J Pall Care 1994;10:57-70. 
30. Holland JC, Cassem EH, Mount BM, Simmons P. Dealing with loss, death, and grief in clinical 
practice: spiritual meaning in oncology.  ASCO Educational Book: American Society of Clinical 
Oncology; 2002:198-205. 
31. Kreitler S. Denial in cancer patients. Cancer Investigation 1999;17:514-34. 
32. Eagly AH, Chaiken S. Cognitive theories of persuasion. In: Berkowitz L, ed. Advances in 
experimental social psychology. San Diego, CA: Academic Press; 1984:267-359. 
33. Petty RE, Cacioppo JT. The elaboration likelihood model of persusasion. In: Berkowitz L, ed. 
Advances in experiemental social psychology. San Diego, CA: Academic Press; 1986:123-205. 
34. Bohner G, Chaiken S, Hunyadi P. The role of mood and message ambiguity in the interplay of 
heuristic and systematic processing. Eur J Soc Psych 1994;24:207-21. 
35. Levine LJ, Burgess SL. Beyond general arousal: Effects of specific emotions on memory. Social 
Cognition 1997;15:157-81. 
36. Niederhoffer KG, Pennebaker JW. Sharing one's story: On the benefits of writing or talking about 
emotional experience. In: Snyder CR, Lopez SJ, eds. Handbook of positive psychology. New York: 
Oxford University Press; 2002. 
37. Cameron LD, Nicholls G. Expression of stressful experiences through writing: effects of a self-
regulation manipulation for pessimists and optimists. Health Psychol 1998;17:84-92. 
38. Greenberg MA, Wortman CB, Stone AA. Emotional expression and physical health: revising 
traumatic memories or fostering self-regulation? J Pers Soc Psychol 1996;71:588-602. 
39. Pennebaker JW, Barger SD, Tiebout J. Disclosure of traumas and health among Holocaust survivors. 
Psychosom Med 1989;51:577-89. 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
29 
 
40. Pennebaker JW, O'Heeron RC. Confiding in others and illness rate among spouses of suicide and 
accidental-death victims. J Abnorm Psychol 1984;93:473-6. 
41. Petrie KJ, Booth RJ, Pennebaker JW, Davison KP, Thomas MG. Disclosure of trauma and immune 
response to a hepatitis B vaccination program. J Consult Clin Psychol 1995;63:787-92. 
42. Stanton AL, Danoff-Burg S, Sworowski LA, et al. Randomized, controlled trial of written emotional 
expression and benefit finding in breast cancer patients. J Clin Oncol 2002;20:4160-8. 
43. Friedrichsen MJ, Strang PM, Carlsson ME. Breaking bad news in the transition from curative to 
palliative cancer care--patient's view of the doctor giving the information. Support Care Cancer 
2000;8:472-8. 
44. Mager WM, Andrykowski MA. Communication in the cancer 'bad news' consultation: patient 
perceptions and psychological adjustment. Psychooncology 2002;11:35-46. 
45. Fogarty LA, Curbow BA, Wingard JR, McDonnell K, Somerfield MR. Can 40 seconds of 
compassion reduce patient anxiety? J Clin Onc 1999;17:371-9. 
46. Chaturvedi SK, Shenoy A, Prasad KM, Senthilnathan SM, Premlatha BS. Concerns, coping and 
quality of life in head and neck cancer patients. Support Care Cancer 1996;4:186-90. 
47. Farrell C, Heaven C, Beaver K, Maguire P. Identifying the concerns of women undergoing 
chemotherapy. Patient Educ Couns 2005;56:72-7. 
48. Heaven CM, Maguire P. The relationship between patients' concerns and psychological distress in a 
hospice setting. Psycho-Oncology 1998;7:502-7. 
49. Parle M, Jones B, Maguire P. Maladaptive coping and affective disorders among cancer patients. 
Psychol Med 1996;26:735-44. 
50. Detmar SB, Aaronson NK, Wever LD, Muller M, Schornagel JH. How are you feeling? Who wants 
to know? Patients' and oncologists' preferences for discussing health-related quality-of-life issues. J 
Clin Oncol 2000;18:3295-301. 
51. Wenrich MD, Curtis JR, Shannon SE, Carline JD, Ambrozy DM, Ramsey PG. Communicating with 
dying patients within the spectrum of medical care from terminal diagnosis to death. Arch Intern 
Med 2001;161:868-74. 
52. Ryan H, Schofield P, Cockburn J, et al. How to recognize and manage psychological distress in 
cancer patients. Eur J Cancer Care (Engl) 2005;14:7-15. 
53. Levinson W, Roter DL, Mullooly JP, Dull VT, Frankel RM. Physician-patient communication. The 
relationship with malpractice claims among primary care physicians and surgeons. JAMA 
1997;277:553-9. 
54. Bertakis KD, Roter D, Putnam SM. The relationship of physician medical interview style to patient 
satisfaction. J Fam Pract 1991;32:175-81. 
55. Roter DL, Hall JA, Katz NR. Relations between physicians' behaviors and analogue patients' 
satisfaction, recall, and impressions. Med Care 1987;25:437-51. 
56. Kaplan SH, Greenfield S, Ware JE, Jr. Assessing the effects of physician-patient interaction on the 
outcomes of chronic disease. Med Care 1989;27:S110-27. 
57. Stewart M, Meredith L, Brown JB, Galajda J. The influence of older patient-physician 
communication on health and health-related outcomes. Clin Geriatr Med 2000;16:25-36. 
58. Butow PN, Dunn SM, Tattersall MH, Jones QJ. Computer-based interaction analysis of the cancer 
consultation. Br J Cancer 1995;71:1115-21. 
59. Epstein RM, Franks P, Fiscella K, et al. Measuring patient-centered communication in patient-
physician consultations: theoretical and practical issues. Soc Sci Med 2005;61:1516-28. 
60. Epstein RM, Street RL, Jr. . Patient-Centered Communication in Cancer Care: Promoting Healing 
and Reducing Suffering. In: Institute NC, ed. Bethesda, MD: ; 2007. 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
30 
 
61. Baile WF, Buckman R, Lenzi R, Glober G, Beale EA, Kudelka AP. SPIKES-A six-step protocol for 
delivering bad news: application to the patient with cancer. Oncologist 2000;5:302-11. 
62. Fischer GS, Tulsky JA, Arnold RM. Communicating a poor prognosis. In: Portenoy RK, Bruera E, 
eds. Topics in Palliative Care. New York: Oxford University Press; 2000. 
63. Larson DG, Tobin DR. End-of-life conversations: evolving practice and theory. JAMA 
2000;284:1573-8. 
64. Parle M, Maguire P, Heaven C. The development of a training model to improve health 
professionals' skills, self-efficacy and outcome expectancies when communicating with cancer 
patients. Soc Sci Med 1997;44:231-40. 
65. Ptacek JT, Eberhardt TL. Breaking bad news. A review of the literature. JAMA 1996;276:496-502. 
66. von Gunten CF, Ferris FD, Emanuel LL. The patient-physician relationship. Ensuring competency 
in end-of-life care: communication and relational skills. JAMA 2000;284:3051-7. 
67. Gillotti C, Thompson T, McNeilis K. Communicative competence in the delivery of bad news. 
Social Science & Medicine 2002;54:1011-23. 
68. Baile WF, Glober GA, Lenzi R, Beale EA, Kudelka AP. Discussing disease progression and end-of-
life decisions. Oncology (Huntingt) 1999;13:1021-31. 
69. von Gunten CF. Discussing hospice care. Journal of Clinical Oncology 2002;20:1419-24. 
70. Lo B, Quill T, Tulsky J. Discussing palliative care with patients. Ann Intern Med 1999;130:744-49. 
71. Butow PN, Kazemi JN, Beeney LJ, Griffin AM, Dunn SM, Tattersall MH. When the diagnosis is 
cancer: patient communication experiences and preferences. Cancer 1996;77:2630-7. 
72. Clark RE, LaBeff EE. Death telling: managing the delivery of bad news. J Health Soc Behav 
1982;23:366-80. 
73. Eden OB, Black I, MacKinlay GA, Emery AE. Communication with parents of children with cancer. 
Pall Med 1994;8:105-14. 
74. Ford S, Fallowfield L, Lewis S. Can oncologists detect distress in their out-patients and how 
satisfied are they with their performance during bad news consultations? Br J Cancer 1994;70:767-
70. 
75. Ford S, Fallowfield L, Lewis S. Doctor-patient interactions in oncology. Soc Sci Med 1996;42:1511-
9. 
76. Emanuel LL, Barry MJ, Stoeckle JD, Ettelson LM, Emanuel EJ. Advance directives for medical 
care--a case for greater use. N Engl J Med 1991;324:889-95. 
77. Layson RT, Adelman HM, Wallach PM, Pfeifer MP, Johnston S, McNutt RA. Discussions about the 
use of life-sustaining treatments: a literature review of physicians' and patients' attitudes and 
practices. J Clin Ethics 1994;5:195-203. 
78. Miles SH, Bannick-Mohrland S, Lurie N. Advance-treatment planning discussions with nursing 
home residents: Pilot experience with simulated interviews. J Clin Ethics 1990;1:108-12. 
79. Miller DK, Coe RM, Hyers TM. Achieving consensus on withdrawing or withholding care for 
critically ill patients. J Gen Int Med 1992;7:475-80. 
80. Tulsky JA, Chesney MA, Lo B. How do medical residents discuss resuscitation with patients? J Gen 
Intern Med 1995;10:436-42. 
81. Tulsky JA, Fischer GS, Rose MR, Arnold RM. Opening the black box: how do physicians 
communicate about advance directives? Ann Intern Med 1998;129:441-9. 
82. Ventres W, Nichter M, Reed R, Frankel R. Do-not-resuscitate discussions: a qualitative analysis. 
Fam Pract Res J 1992;12:157-69. 
83. Ventres W, Nichter M, Reed R, Frankel R. Limitation of medical care: an ethnographic analysis. J 
Clin Ethics 1993;4:134-45. 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
31 
 
84. Maguire P, Faulkner A, Booth K, Elliott C, Hillier V. Helping cancer patients disclose their 
concerns. Eur J Cancer 1996;32A:78-81. 
85. Maguire P. Improving communication with cancer patients. Eur J Cancer 1999;35:1415-22. 
86. Heaven CM, Maguire P. Disclosure of concerns by hospice patients and their identification by 
nurses. Pall Med 1997;11:283-90. 
87. Detmar SB, Muller MJ, Wever LD, Schornagel JH, Aaronson NK. The patient-physician 
relationship. Patient-physician communication during outpatient palliative treatment visits: an 
observational study. JAMA 2001;285:1351-7. 
88. Koropchak CM, Pollak KI, Arnold RM, et al. Studying communication in oncologist-patient 
encounters: the SCOPE Trial. Palliat Med 2006;20:813-9. 
89. Pollak KI, Arnold RM, Jeffreys AS, et al. Oncologist communication about emotion during visits 
with patients with advanced cancer. J Clin Oncol 2007;25:5748-52. 
90. Bandura A. Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, 
NJ, USA: Prentice Hall, Inc; 1986. 
91. Prochaska JO, DiClemente CC. Self change processes, self efficacy and decisional balance across 
five stages of smoking cessation. Prog Clin Biol Res 1984;156:131-40. 
92. Cabana MD, Rand CS, Powe NR, et al. Why don't physicians follow clinical practice guidelines? A 
framework for improvement. JAMA 1999;282:1458-65. 
93. Butow PN, Brown RF, Cogar S, Tattersall MH, Dunn SM. Oncologists' reactions to cancer patients' 
verbal cues. Psychooncology 2002;11:47-58. 
94. Brown JB, Boles M, Mullooly JP, Levinson W. Effect of clinician communication skills training on 
patient satisfaction. A randomized, controlled trial. Ann Intern Med 1999;131:822-9. 
95. Lipkin M, Kaplan C, Clark W, Novack DH. Teaching medical interviewing: The Lipkin Model. In: 
Lipkin M, Putnam SM, Lazare A, eds. The Medical Interview. New York: Springer-Verlag; 
1995:422-35. 
96. Roter DL, Hall JA, Kern DE, Barker LR, Cole KA, Roca RP. Improving physicians' interviewing 
skills and reducing patients' emotional distress. A randomized clinical trial. Arch Intern Med 
1995;155:1877-84. 
97. Levinson W, Roter D. The effects of two continuing medical education programs on communication 
skills of practicing primary care physicians. J Gen Intern Med 1993;8:318-24. 
98. Worstell M. Asthma: individual patient perspective and current unmet needs. Clin Exp Allergy 
2000;30 Suppl 1:11-5. 
99. Baile WF, Kudelka AP, Beale EA, et al. Communication skills training in oncology. Description and 
preliminary outcomes of workshops on breaking bad news and managing patient reactions to illness. 
Cancer 1999;86:887-97. 
100. Jenkins V, Fallowfield L. Can communication skills training alter physicians' beliefs and behavior in 
clinics? J Clin Onc 2002;20:765-9. 
101. Fallowfield L, Jenkins V, Farewell V, Saul J, Duffy A, Eves R. Efficacy of a Cancer Research UK 
communication skills training model for oncologists: a randomised controlled trial. Lancet 
2002;359:650-6. 
102. Fallowfield L, Jenkins V, Farewell V, Solis-Trapala I. Enduring impact of communication skills 
training: results of a 12-month follow-up. Br J Cancer 2003;89:1445-9. 
103. Alexander SC, Keitz SA, Sloane R, Tulsky JA. A controlled trial of a short course to improve 
residents' communication with patients at the end of life. Acad Med 2006;81:1008-12. 
104. Back AL, Arnold RM, Tulsky JA, Baile WF, Fryer-Edwards KA. Teaching communication skills to 
medical oncology fellows. J Clin Oncol 2003;21:2433-6. 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
32 
 
105. Back AL, Arnold RM, Baile WF, et al. Efficacy of communication skills training for giving bad 
news and discussing transitions to palliative care. Arch Intern Med 2007;167:453-60. 
106. Skinner CS, Pollak KI, Farrell D, Olsen MK, Jeffreys AS, Tulsky JA. Use of and reactions to a 
tailored CD-ROM designed to enhance oncologist-patient communication: the SCOPE trial 
intervention. Pt Educ Couns 2009;77:90-6. 
107. Tulsky JA, Arnold RM, Alexander SC, et al. Enhancing communication between oncologists and 
patients with a computer-based training program: a randomized trial. Ann Intern Med 2011;155:593-
601. 
108. Hawkins RE, Lipner RS, Ham HP, Wagner R, Holmboe ES. American Board of Medical Specialties 
Maintenance of Certification: theory and evidence regarding the current framework. The Journal of 
continuing education in the health professions 2013;33 Suppl 1:S7-19. 
109. Lipner RS, Hess BJ, Phillips RL, Jr. Specialty board certification in the United States: issues and 
evidence. The Journal of continuing education in the health professions 2013;33 Suppl 1:S20-35. 
110. Francis LK. Learning to listen: a fellow's experience. J Clin Oncol 2006;24:3209-10. 
111. Dyer N, Sorra JS, Smith SA, Cleary PD, Hays RD. Psychometric properties of the Consumer 
Assessment of Healthcare Providers and Systems (CAHPS(R)) Clinician and Group Adult Visit 
Survey. Med Care 2012;50 Suppl:S28-34. 
112. Mukherjee S, Rodriguez HP, Elliott MN, Crane PK. Modern psychometric methods for estimating 
physician performance on the Clinician and Group CAHPS survey. Health Serv Outcomes Res 
Method 2013;13:109-23. 
113. Suchman AL, Markakis K, Beckman HB, Frankel R. A model of empathic communication in the 
medical interview. JAMA 1997;277:678-82. 
114. Smith RC, Hoppe RB. The patient's story: integrating the patient- and physician-centered 
approaches to interviewing. Ann Intern Med 1991;115:470-7. 
115. Tulsky JA. Doctor-patient communication issues. In: Cassel CK, Leipzig RM, Cohen HJ, Larson 
EB, Meier DE, Capello CF, eds. Geriatric Medicine. 4th ed. New York: Springer; 2003:287-98. 
116. Hintze, J. (2011). PASS 11. NCSS, LLC. Kaysville, Utah, USA. www.ncss.com. 
117. Chow, SC, Shao, J, Wang, H. Sample Size Calculations in Clinical Research. Boca Raton, FL: CRC 
Press, Taylor & Francis Group; 2003. 
118. Donner, A. and Klar, N. 1996. 'Statistical Considerations in the Design and Analysis of Community 
InterventionTrials.' The Journal of Clinical Epidemiology, Vol. 49, No. 4, 1996, pages 435-439. 
119. Donner, A. and Klar, N. 2000. Design and Analysis of Cluster Randomization Trials in Health 
Research. Arnold. London. 
120. Arnold, GK, Baranowski, RA, Duhigg, L. (2014) Can Planned Visit Scores Predict Practice                           
Improvements in Patient Experience Outcomes? A Meta-Regression Study. In JSM Proceedings Health 
Policy Statistics Section, Alexandria, VA, American Statistical Association, 2505-2516. 
121. Hsieh, F.Y., Block, D.A., and Larsen, M.D. 1998. 'A Simple Method of Sample Size Calculation for 
Linear and Logistic Regression', Statistics in Medicine, Volume 17, pages 1623-1634. 
122. Signorini, David. 1991. 'Sample size for Poisson regression', Biometrika, Volume 78, 2,  pages 446-
450. 
123. Dyer, N, Sorra, JS, Smith, SA, Cleary, PD, Hays, RD. (2012), “Psychometric Properties of the 
Consumer Assessment of Healthcare Providers and Systems CAHPS® Clinician and Group Adult 
Visit Survey” Med Care 2012;50: S28–S34. 
124. Murray DM. Design and Analysis of Group Randomized Trials. NY Oxford University Press; 1998 
125 Little, RJA, Rubin, DB. Statistical Analysis with Missing Data. Hoboken, NJ: Wiley & Sons; 2002 

 
 
Improving Communication Between Cancer Patients & Oncologists: 05/07/18 
 
 
 
 
33 
 
126. Fitzmaurice GM, Laid, NM, Ware, JH. Applied Longitudinal Analysis. Hoboken, NJ: Wiley & 
Sons; 2004 
127. Hedeker D, Gibbons RD. Longitudinal Data Analysis. Hoboken, NJ: Wiley & Sons; 2006 
128. Benjamini, Y. and Hochberg, Y. (1995), “Controlling the False Discovery Rate: A Practical and 
Powerful Approach to Multiple Testing,” Journal of the Royal Statistical Society, B, 57, 289–300. 
129  Block SD, Clark-Chiarelli N, Peters AS, Singer JD. Academia's chilly climate for primary care. JAMA 
1996;276:677-82.