F1000Research 2015, 4:639 Last updated: 16 MAY 2019

CORRESPONDENCE

Double blinding requirement for validity claims in
cognitive-behavioral therapy intervention trials for major
depressive disorder. Analysis of Hollon S, et al., Effect ofcognitive therapy with antidepressant medications vsantidepressants alone on the rate of recovery in major
depressive disorder: a randomized clinical trial [version 1; peer
review: 2 approved]
Douglas Berger
Meguro Counseling Center, Tokyo, Japan

published: 27 Aug 2015, 4:639 (v1 First
https://doi.org/10.12688/f1000research.6954.1)
Latest published: 27 Aug 2015, 4:639 (https://doi.org/10.12688/f1000research.6954.1)

Abstract
This paper will focus on problems in the inability to double-blind
cognitive-behavioral therapy (CBT) studies for major depressive disorder(MDD), and provides an analysis of a recently published study to show how
this problem can lead to faulty conclusions.
A study by Hollon et al. published in JAMA Psychiatry that compared anantidepressant medication-only arm with a combined CBT/antidepressantarm concluded that the cognitive therapy/antidepressant combination
enhanced the recovery rates compared with antidepressant alone, and thatthe magnitude of this increment nearly doubled for patients with more
severe depression.
We propose that for subjects with greater severity, there could have beenboth antidepressant efficacy as well as more hope and expectation in thegroup who knew they had received combined cognitive therapy/medication,
leading to an erroneous conclusion of greater efficacy for the combinedgroup. The large subject number in this study could easily lead to anerroneous finding on statistical testing as a small amount of bias in thesubjects adds-up.

Open Peer Review
Reviewer Status
Invited Reviewers

12

reportreportversion 1
published
27 Aug 2015

1 Rebecca Graham, Prince of Wales Hospital,
Randwick, Australia
2 Arif Khan, Northwest Clinical Research Center,
Bellevue, USA
Any reports and responses or comments on thearticle can be found at the end of the article.

We opine that the conclusions of unblind CBT outcome research inconditions with subjective endpoints such as MDD need to be given withgreat caution. The validity of CBT (and its derivatives such as dialecticalbehavioral therapy) for indications other than MDD is also part of a larger
problem in the inability to blind outcome studies for these interventions.

Page 1 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

Keywords
psychotherapy , cognitive-behavioral therapy (CBT) , outcome studies ,blinding , clinical trials

Corresponding author: Douglas Berger (doug@japanpsychiatrist.com)
Competing interests: No competing interests were disclosed. The author has no financial interests, activities, relationships, and affiliations otherthan those affiliations listed in the title page of the manuscript. There was no data collected or analyzed for this paper.
Grant information: The author declared that no grants were involved in supporting this work
Copyright: © 2015 Berger D. This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which
permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
How to cite this article: Berger D. Double blinding requirement for validity claims in cognitive-behavioral therapy intervention trials for
major depressive disorder. Analysis of Hollon S, et al., Effect of cognitive therapy with antidepressant medications vsantidepressants alone on the rate of recovery in major depressive disorder: a randomized clinical trial [version 1; peer review: 2
approved] F1000Research 2015, 4:639 (https://doi.org/10.12688/f1000research.6954.1)
First published: 27 Aug 2015, 4:639 (https://doi.org/10.12688/f1000research.6954.1)

Page 2 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

The validity of cognitive-behavioral therapy (CBT) efficacy for
major depressive disorder (MDD) is widely accepted and is basedlargely on clinical intervention studies of CBT in MDD. However,clinical trials for CBT cannot be carried out under double-blindconditions as would be required of pharmacotherapy (or othersomatic therapies), thus the rigor of CBT interventional studies
is quite different from those modalities that can be studied under
double-blinded conditions1,2.
Treatment allocation cannot be blinded in CBT studies because thesubjects have to actively participate in cognitive restructuring tasks.
More than just saying a study was “blinded”, absolute concealmentof what treatment was allocated is crucial in order to avoid bias3.
CBT trials are sometimes stated to be, “single-blind” because thepersons who rate the symptoms that subjects report are blind to
the treatment allocation of the subject. The term “single-blind”,
however, should be used with caution as single-blind is definedas the condition when subjects are blind, not the raters4. Blind (or“masked”) raters only record whatever bias may be in the subjective
reports of the subjects that can be swayed by the unblinded conditions. Emphasizing that raters are blind in a CBT study can distractfrom the issue that subjects and treaters are not blind.
Allocation concealment is crucial for indications with subjective
outcomes as in MDD3. During a clinical trial, subjects with MDD
report changes in the severity of subjective depressive symptoms
that may be influenced by an expectation or hope for improvement3.
Only interventional studies for indications with objective endpoints
can ignore potential bias from lack of blinding. For example,
mortality rates, MI incidence, stroke, etc. where random error issmall1. In this line, a meta analysis of CBT trials that controlled forblinding found treatment effects to be small in MDD5.
However, studies continue to report positive results of unblinded
trials without voicing strong caution on the validity of the results.
Hollon et al. in the October 2014 issue of JAMA Psychiatry compared an antidepressant medication only arm with a combined cognitive therapy/antidepressant arm6. All the subjects who received
antidepressants did so under unblinded conditions. The cognitive therapy subjects and their treaters were also unblind to thetreatment given. The study concluded that the cognitive therapy/
antidepressant combination enhanced the rate of recovery comparedwith antidepressant alone, and that the magnitude of this increment
nearly doubled for patients with more severe depression with little
evidence of benefit for patients with less severe MDD. Only one
line at the end of the discussion noted that the unblinded conditions
could be a limitation.
An alternative conclusion could just as easily be that patients withgreater severity MDD may have included more patients with a
medication-responsive depression7. For those subjects with greater
severity, there could have been both antidepressant efficacy aswell as more hope and expectation in the group who knew they
had received combined cognitive therapy/medication leading to anerroneous conclusion of greater efficacy for the combined group.
A large sample size (N) as in this study is not necessarily a signof robust results. A large N can create a significant finding on statistical testing as a small amount of bias in the subjects adds-up1.

Our alternative conclusion may also be incorrect, the important
issue is that the lack of allocation concealment in the study design
does not allow any valid conclusion to be made either way. The
antidepressant in each arm of the study provides the same amountof hope and expectation; the CBT arm has the added potential forbias from hope and expectation.
In addition, combining and comparing antidepressants that havemarket approval based on double-blinded placebo controlled outcome research with CBT, heretofore never studied under double-, or
single-blinded conditions, in the same unblinded study is a serious
problem. Handicapping one intervention group (antidepressants without the double-blinded placebo control needed for proof of efficacy),
while providing advantage to another intervention group (unblinded
CBT with no psychotherapy placebo which allows bias in one arm)
which is then mixed with the handicapped group, confounds thestudy conditions and invalidates the design logic of a clinical trial.
To be sure, interventional studies for somatic therapies such as
medications may also have elements of allocation non-concealment
requiring caution in their interpretation. While medications can feasibly be blinded, side-effects may expose a subject to the fact thatthey are in the active-drug arm of a study. An exit analysis on the
proportion of subjects in a study that correctly guessed the treatment
arm they were in should be done, and the results of any study in
an indication with subjective endpoints such as MDD that has evidence of unblinding should be suspect to have bias. Psychotherapy
treatment, on the other hand, is virtually impossible to hide fromthe subject who is openly given the treatment. Whether medication, psychotherapy, or other intervention, no valid scientific assessment of efficacy can be made if a hurdle such as double-blinding
in the study design of an indication with subjective endpoints is not
rigorously implemented.
Authors must state clearly when an intervention cannot be studied with rigor, and conclusions need to be given with great caution
when studies with subjective endpoints are unblinded. There is no
regulatory authority like the FDA to review and approve a psychotherapeutic intervention for MDD, so that both professionals andsociety at large alike are dependent on the sound-bite conclusions
made by authors and commentators on the results reported.
The critical problem of the inability to double-blind CBT clinical trials for MDD requires further evaluation by research groups
who do not have a vested interest in CBT or related therapies. The
validity of CBT (and its derivatives such as dialectical behavioral
therapy) for indications other than MDD is part of a larger problem
in the inability to blind outcome for these interventions.

Competing interests
No competing interests were disclosed. The author has no financial
interests, activities, relationships, and affiliations other than thoseaffiliations listed in the title page of the manuscript. There was nodata collected or analyzed for this paper.
Grant information
The author(s) declared that no grants were involved in supporting
this work.
Page 3 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

References
1.

Piantadosi S: Clinical Trials: A Methodologic Perspective, Second Edition. New
York: Wiley-Interscience, 2005.
Publisher Full Text

2.

Hanrahan C, New JP: Antidepressant Medications: The FDA-Approval Process
and the Need for Updates. Ment Health Clin. 2014; 4(1): 11–16.
Publisher Full Text

3.

Schulz KF, Grimes DA: Blinding in randomised trials: hiding who got what.
Lancet. 2002; 359(9307): 696–700.
PubMed Abstract | Publisher Full Text

4.

Friedman LM, Furgerg CD, DeMets DL: Fundamentals of Clinical Trials, Third
Edition. Springer; 1998.
Reference Source

5.

Lynch D, Laws KR, McKenna PJ: Cognitive behavioural therapy for major
psychiatric disorder: does it really work? A meta-analytical review of wellcontrolled trials. Psychol Med. 2010; 40(1): 9–24.
PubMed Abstract | Publisher Full Text

6.

Hollon SD, DeRubeis RJ, Fawcett J, et al.: Effect of cognitive therapy with antidepressant
medications vs antidepressants alone on the rate of recovery in major depressive
disorder: a randomized clinical trial. JAMA Psychiatry. 2014; 71(10): 1157–1164.
PubMed Abstract | Publisher Full Text | Free Full Text

7.

Khan A, Leventhal RM, Khan SR, et al.: Severity of depression and responseto antidepressants and placebo: an analysis of the Food and Drug
Administration database. J Clin Psychopharmacol. 2002; 22(1): 40–45.
PubMed Abstract | Publisher Full Text

Page 4 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

Open Peer Review
Current Peer Review Status:
Version 1
Reviewer Report 09 May 2016

https://doi.org/10.5256/f1000research.7489.r13511© 2016 Khan A. This is an open access peer review report distributed under the terms of the Creative Commons
Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original
work is properly cited.

Arif Khan
Northwest Clinical Research Center, Bellevue, WA, USA
Dr. Berger has taken to task a sacred cow in our field. Psychotherapy works and any challenges to it
amount to sacrilegious position! Unfortunately for the dogmatic, Dr. Berger is right. It is not possible totruly 'blind' a psychotherapy study.
Simply put, the model of blinded raters is limited as both the patient and therapist are not blinded.
Whether one likes it or not, the enthusiasm and conviction of a therapist is more than porous. The
Freudian model is that a cigar smoking therapist begets a cigar smoking patient.
In the same note the sham therapist's conviction and enthusiasm (or a lack of it) will also be picked up by
the patients. We have shown that the level of blinding does change expectations and expectations for thepsychotherapy + pharmacotherapy results in the expected increased benefit.1
Having said this, it would be foolish of me to suggest that psychotherapy has no value. After all, practiceof medicine, psychiatry and psychology is intrinsically and intricately based on human interactions. So,
training and practicing of psychotherapy is essential, just like saying you have to know anatomy before
you start surgery. It is an essential skill in humanistic sciences. However, just like proving you need toknow anatomy before you cut somebody up surgically cannot really be tested using current models andthe same applies to psychotherapy.
In this context, it is important to note that different types of psychotherapy are better than other ones. Thisis like testing a surgical knife for different surgeries. All the psychotherapies (if practiced well) are
effective just like all knives are effective.
It is only a poor clinician who tries to treat a patient without the understanding and application of
psychotherapy principles to distressed patients. Here, I would like to note that Dr. Berger is a known
psychotherapist. Thus, his position is courageous and to be lauded.
References
1. Khan A, Faucett J, Lichtenberg P, Kirsch I, Brown WA: A systematic review of comparative efficacy oftreatments and controls for depression.PLoS One. 2012; 7 (7): e41778 PubMed Abstract | Publisher Full
Page 5 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

treatments and controls for depression.PLoS One. 2012; 7 (7): e41778 PubMed Abstract | Publisher Full
Text
Competing Interests: No competing interests were disclosed.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that
it is of an acceptable scientific standard.
Reviewer Report 26 October 2015

https://doi.org/10.5256/f1000research.7489.r10880© 2015 Graham R. This is an open access peer review report distributed under the terms of the Creative Commons
Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original
work is properly cited.

Rebecca Graham
Black Dog Institute, Hospital Road, Prince of Wales Hospital, Randwick NSW, Australia 2031
Overall, a reasonably well-written summary of the complexities evident in CBT intervention trials for MDD.
The only suggestion I would make is that the grammar should be checked carefully prior to publication especially the correct use of commas. Other than that - would agree with indexation.
Competing Interests: No competing interests were disclosed.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that
it is of an acceptable scientific standard.

Comments on this article
Version 1

Author Response 12 Oct 2015

Douglas Berger, Meguro Counseling Center, Tokyo, Japan
Directive psychotherapies are not more effective than non-directive psychotherapies whencontrolled for researcher allegiance.
We noticed that Dr. Hollon is actually one of the authors of the paper we quoted from 2012, that concluded
that directive psychotherapies were not more effective than non-directive psychotherapies when controlled
for researcher allegiance, and that the majority of the effect of therapy for adult depression is realized by
non-specific factors1.
This is the abstract of this paper [our bolding]:“The effects of non-directive supportive therapy (NDST) for adult depression have been examined in aconsiderable number of studies, but no meta-analysis of these studies has been conducted. We selected
Page 6 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019considerable number of studies, but no meta-analysis of these studies has been conducted. We selected
31 studies on NDST from a comprehensive database of trials, examining psychotherapies for adultdepression, and conducted meta-analyses in which NDST was compared with control groups, otherpsychotherapies and pharmacotherapy. We found that NDST is effective in the treatment of depression in
adults (g = 0.58; 95% CI: 0.45–0.72). NDST was less effective than other psychological treatments(differential effect size g = − 0.20; 95% CI: − 0.32 to − 0.08, p < 0.01), but these differences were nolonger present after controlling for researcher allegiance. We estimated that extra-therapeutic
factors (those processes operating in waiting-list and care-as-usual controls) were responsible for 33.3%of the overall improvement, non-specific factors (the effects of NDST compared with control groups) for
49.6%, and specific factors (the effects of NDST compared with other therapies) for 17.1%. NDST has aconsiderable effect on symptoms of depression. Most of the effect of therapy for adult depression isrealized by non-specific factors, and our results suggest that the contribution of specific effects
is limited at best.”
Although Hollon and his coauthors are admitting in the above paper that the unblinded nature of therapists
who have some allegiance to the therapy they are trained in can be the cause of positive findings, Hollon
et al. now opine in their comment to us that blinding is not important for validation of CT efficacy, and that
CT is superior to non-directive supportive therapies by referencing papers published the following year in
2013.2,3
We are not sure why/how Dr. Hollon views these spins of his own conclusions over this short period of one
year, nor how Dr. Hollon can state that psychotherapy studies do not require double-blinding to filter out
bias.
We also noted Parker, et al.4 did a systematic literature search and could not find evidence that CBT was
distinctively superior than other psychotherapies for major depression.
However, regardless of the issue of directiveness of a therapy, there is no way to get around the problem
of not being able to blind the subjects or the treaters in any psychotherapy trial, directive or not.
That researcher allegiance is a cause of bias as concluded by a paper by Dr. Hollon himself, it is clear that
conclusions must be seriously tempered in CBT trials where neither subjects nor treaters can be blinded,
and where the majority of treaters trained in CBT would likely believe in CBT if they continued to beproviders of this treatment.

References
1. Cuijpers P, Driessen E, Hollon SD, van Oppen P, Barth J, Andersson G: The efficacy of non-directive
supportive therapy for adult depression: A meta-analysis. Clin Psychol Rev.2012; 32 (4): 280-291 PubMed
Abstract | Publisher Full Text | Reference Source
2. Barth J, Munder T, Gerger H, Nüesch E, Trelle S, Znoj H, Jüni P, Cuijpers P: Comparative efficacy ofseven psychotherapeutic interventions for patients with depression: a network meta-analysis. PLoS Med.
2013; 10 (5): e1001454 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
3. Braun SR, Gregor B, Tran US: Comparing bona fide psychotherapies of depression in adults with twometa-analytical approaches. PLoS One. 2013; 8 (6): e68135 PubMed Abstract | Free Full Text | Publisher
Full Text | Reference Source
4. Parker G, Fletcher K: Treating depression with the evidence-based psychotherapies: a critique of theevidence. Acta Psychiatr Scand. 2007; 115 (5): 352-359 PubMed Abstract | Reference Source

Page 7 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

Competing Interests: There are no competing interests.

Author Response 01 Sep 2015

Douglas Berger, Meguro Counseling Center, Tokyo, Japan
Blinding and blind placebo are clearly crucial to elucidate efficacy in disorders with subjective
endpoints.
We appreciate that Hollon et al. share our preference for double-blind controls. However, we areconcerned that their logic of how to conduct and interpret the results of a clinical trial for an indication withsubjective endpoints that can not be single- or double-blinded (and thus can not have a blind placebogroup) does not describe the scientific needs of these kinds of trials adequately. We will address some of
the most important points they describe here (we use CT (Cognitive Therapy) and CBT interchangeably tomean the same type of psychotherapy as in our article above in this comment; and Hollon et al. to refer tothe paper discussed in our article above, and/or the comment to our paper by Hollon et al. above).
It is imperative that we repeat that double blinding is crucial for indications with subjective outcomes as in1
MDD . Only interventional studies for indications with objective endpoints can ignore potential bias from2lack of blinding. For example, mortality rates, MI incidence, stroke, etc. where random error is small . This
study on bias concluded that unblinded randomized clinical trials tended to be biased to positive effects if
3
the outcomes were subjective as opposed to objective .
To add additional important comment to our paper:
1. non-effective drugs and placebo pills can both show an average 30% improvement in depression
4
scores compared to baseline (not due to improvements from waiting) . Called a, “placebo effect”
5
this is suspected to be due to the subject’s hope and expectation for improvement .
6
2. So-called “psychological placebos” can be just as effective as standard psychotherapies in MDD .

It is clear that because of the propensity of persons diagnosed with depression to improve with
non-specific factors (i.e., receiving some intervention even if placebo pill or therapy, relationship with acaretaker, participation in a trial, etc.), enhanced improvement can be reported by subjects if there is evena small amount of bias to a specific intervention given in an unblinded fashion. Improvement from this bias
will then be reported by the subject to the rater and can add up such that they will become significant with alarge N as was the case in the study by Hollon et al. discussed in our paper.
We are at a loss as to how Hollon et al., can argue their way out of this conundrum and make claims ofefficacy without calling for serious hesitation in the interpretation of results from CBT trials such as in
Hollon et al.
Regarding their points.
Point 1: Single-Blinding: definitions and misuse of terms.
Hollon et al. continue to call CBT trials “single-blind” in their comment. We noted in our paper that7

“single-blind” is defined as the condition when subjects are blind, not the treaters or only the raters .
Page 8 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 20197

“single-blind” is defined as the condition when subjects are blind, not the treaters or only the raters .
See this reference in the American Psychological Association methodology data base, under, “Treatment
Outcome/Clinical Trial”: “It is single-blind when the participants do not know which treatment group they
have been assigned to”, (http://www.apa.org/pubs/databases/training/method-values.aspx, Accessed
August 29th, 2015).
Using the term “single-blind” for studies that are not actually single-blind under this definition is not using
the proper definition of this term and seems to be trying to suggest there is more rigor and control in a CBTstudy than is actually there.
The definition for single-blind is not met for a CBT trial as Hollon et al. acknowledge, so we are confused
why they continue to use this term for trials with unblinded subjects. None of the trials they reference havea single-blind psychotherapy arm, and thus they can neither have a blind psychotherapy placebo
comparison group. All the subjects and therapists know the psychotherapy provided. As we also noted in
our paper, you can not logically compare blinded medications with unblinded psychotherapy and makeany logical conclusions. The design of these arms are completely different, and as clearly reference in ourpaper, unblinded subjects in a study with subjective endpoints can not eliminate bias.
Even if Hollon et al. wish to redefine “single-blind” as when only raters are blind as in their study (or any
study of CBT), this doesn’t change the fact that blind raters only record whatever bias may be in thesubjective reports of the subjects that can be swayed by the unblinded conditions. Emphasizing that raters
are blind in a CBT study can distract from the fact that this does not change the potential for bias in thesubject-treater system.
Point 2: Expectancy effects: uncontrolled data, and post hoc analyses.
Their counter to our alternative interpretation of their results that , “It strains the imagination to, for example,
attempt to come up with an explanation as to why expectancy effects should be limited to the third of thesample that was recurrent and non-chronic, but more severe”, is confusing because it is extremely easy to
imagine the logic pitfalls in making associations here.
First and foremost is that the data is based on unblinded/non-placebo control conditions in subjective
endpoints. Hollon et al. are trying to extract some kind of association from data born from uncontrol so that
this is data on shaky ground (see the references in our paper).
Next, Hollon et al. have made a post hoc analysis on multiple variables (severity, acute vs chronic,
8
recurrent or not, etc.) and thus have inflated the possibility of a type I error rate .
This is because the further one looks for associations the more likely something will be found. In otherwords, each time an association in the data is considered, a statistical test is effectively performed. This
inflates the total number of statistical tests needed and p-values need to be adjusted, which we did notsee. Hollon et al. only did an “exploratory analysis” to determine whether severity and chronicity
contributed independently to the increments observed. It is necessary to label these analyses as post hoc
to avoid misleading readers that they are statistically robust when they are not.
The problems in multiple comparisons is common in the use of “data dredging” (also called “data mining”),
to uncover patterns in data that can be presented as statistically significant, without first devising a specific9
hypothesis as to the underlying causality .

Page 9 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

The authors tried to justify the post hoc analysis problem by using a Bonferroni correction. The traditional
Bonferroni tends to lack power, type II error rates are too high for individual tests, and it overcorrects forType I error. Other modified Bonferroni approaches are available (not specified as done by the authors),
but still do not fully get around the post hoc p value problem (Portland University Data Analysis Course,
http://www.upa.pdx.edu/IOA/newsom/da1/ho_posthoc.doc, Accessed September 1, 2015).
It is hard for us to comprehend how Hollon et al. can call their post hoc subgroup analysis “exploratory” inthe body of their text but yet claim these subgroups were clearly part of the efficacy of cognitive therapy intheir Abstract conclusion: “Cognitive therapy combined with ADM treatment enhances the rates ofrecovery from MDD relative to ADMs alone, with the effect limited to patients with severe, nonchronic
depression”.
Regardless of statistics, the Hollon et al. data is born from unblinded and thus uncontrolled study design
so that we opine this data should not be acceptable to begin with. The FDA requires all antidepressants to10
have done some double-blind placebo controlled studies for approval for this reason .
Point 3: Improved care for patients: The risks to claim improved care from unproveninterventions are serious.
If an intervention is touted as efficacious in spite of having only unblinded and no placebo control, as wehave outlined it is for CBT studies of depression with subjective endpoints, it is risky that patients withdepression will seek out this intervention in place of treatments that have much more robust double-blindand placebo controlled data with approvals from a regulatory agency such as the FDA. These patients may
sustain greater morbidity and mortality, and the financial costs to society can be large.
While few U.S. psychiatrists are unfamiliar with Osheroff v. Chestnut Lodge, we feel it is important enough11
to cite an abstract by Klerman in full here . Hollon et al., will undoubtedly opine that CBT is a much more
rigorously studied psychotherapy than the therapy Osheroff received, however, our paper, and otherpapers referenced below (references 13, 14, 18 discussed in Point 4 below), have presented strong logic
to question the entire basis of CBT efficacy studies. We strongly agree with Klerman below that the,“…right of the patient to effective treatment and that treatments whose efficacy has been demonstrated
have priority over treatments whose efficacy has not been established”.
11

Abstract from Klerman :“Although Osheroff v. Chestnut Lodge never reached final court adjudication, the case generated
widespread discussion in psychiatric, legal, and lay circles. The author served as a consultant to Dr.
Osheroff and testified that Chestnut Lodge failed to follow through with appropriate biological treatment for
its own diagnosis of depression, focusing instead on Dr. Osheroff's presumed personality disorderdiagnosis and treating him with intensive long-term individual psychotherapy. The author suggests that thiscase involves the proposed right of the patient to effective treatment and that treatments whose efficacy
has been demonstrated have priority over treatments whose efficacy has not been established.”
Point 4: Listing of the many efficacy studies for CBT.
Hollon et al. list numerous studies they have done to show that CBT is efficacious. The logic here is, there
are many studies with lack of single- or double-blinding control and blind placebo, these studies haveshown CBT to be efficacious, thus, the data born from these studies even if not born from double-blinding
control and blind placebo is robust; i.e., data from many studies with this design problem makes up forPage 10 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

control and blind placebo is robust; i.e., data from many studies with this design problem makes up foreach one of them having this design problem individually. The trouble in the logic here should be obvious.
There was no blind psychotherapy placebo, and no double-blind, nor single-blind study (as defined above)in any of the studies listed. Basing conclusions on data born from uncontrolled (unblinded and no placebo)
circumstances in subjective endpoints, and then making absolute conclusions from this, and furthermore,comparing this data to medication trials born from double-blind placebo controlled studies is a tortuousargument because the data is born from such widely different level of control.
Basing results on data born without single- or double blind controls and blind placebo fits with Ioannidis’
9Corollary 4 : “The greater the flexibility in designs, definitions, outcomes, and analytical modes in ascientific field, the less likely the research findings are to be true.”
12

The subgroup analysis Hollon et al., noted in their reference #7 was also a post hoc analysis . The
authors in this study even went to great pains to take some “middle ground” analysis in order to avoid
conducting “…approximately 150 hypothesis tests, from which we could have expected to find several
statistically significant results by chance”. Reference #7 was also an analysis of an unblinded, non-placebo
controlled psychotherapy study, and again breaking clinical trial design logic by comparing blinded
medication therapy with unblinded CBT. While it is obvious that we must be very careful in concludingmuch from these kinds of results, Hollon et al., continue to push conclusive remarks in their comment toour article.
There is considerable and growing concern that CBT is not an effective treatment for many conditions it
touts itself to be. We noted the Lynch et al. meta analysis of CBT trials that controlled for blinding and13
placebo in our paper . The results of this large meta analysis showed that CBT fared no better thannon-specific control interventions in the treatment of schizophrenia and did not improve relapse rates, CBT
showed no effect in prevention of bipolar disorder episodes, and only small treatment effects were seen in
14
studies of MDD. We also discuss Parker below who found a low evidence base for the specificity andsuperiority of CBT over other psychotherapies.
Regarding the point that “cognitive therapy is about as efficacious as medications when adequately
implemented in the acute phase of treatment and it has an enduring effect that medications cannot match”
15(their reference # 8) . Long-term or prevention studies of CBT can still only be unblinded studies. CBT
more clearly requires subject participation then a non-directive approach and is provided by trained
therapists in an open fashion. It is obvious that bias potential is not removed from these studies. We havedescribed above the obvious flaw of logic to compare medication arms or studies carried-out withdouble-blind placebo with unblinded CBT outcome results because the design of these methods are so
different.
Regarding Hollon et al.’s comment that , “In this context, findings that CT is superior to non-directive
supportive therapies (their reference #s 9,10) suggest the intervention is more than just a placebo.”[ref-16-17]

A meta-analysis found no differences between directive or non-directive psychotherapy intervention when18
controlled for researcher allegiance, and that most of therapy effects came from by non-specific factors.
14

In addition, Parker, et al. did a systematic literature search and concluded, “There is no high level
evidence base suggesting or identifying CBT and IPT [Interpersonal Psychotherapy] as distinctively
superior than other psychotherapies for major depression. Claiming that they are evidence based,
Page 11 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019superior than other psychotherapies for major depression. Claiming that they are evidence based,
implicating or stating their superiority, and formalizing such differential status in therapeutic guidelines isdisingenuous at best and risks distorting the value of an evidence based approach to Psychiatry. The
clinical implications of over-selling CBT and IPT as specific and superior treatments for depression impact
on both patients and practitioners”.
Remember, no study comparing a non-directive therapy to CBT can be blinded. The subjects and CBTpractitioners would know they received CBT, and any preference for a directive approach would not be
filtered by the unblinded nature of these studies.
Point 5: Use of study in Astronomy and Evolutionary Biology as an example that double-blinding
and placebo control are unnecessary aspects of study design to claim efficacy in an interventionof a human disorder with subjective endpoints.
Hollon et al. have confused the use of data collection to make theories in natural science vs. the making of
efficacy claims in a clinical trial of a human condition that has subjective endpoints. The argument about
natural science theory making is not applicable to the conduct of clinical trial that is comparing the outcomeof an intervention vs. some control group and then making efficacy claims. Natural scientists do not claimthat their conclusions or theories should be used to treat a human disorder with subjective endpoints. Thisis diversionary logic that is extremely hard for us to understand in the setting of a discussion about aclinical trial of a human disorder.
Point 6: Competing Interests, potential researcher bias because of grants and academic
interests.
Regarding potential researcher bias, the paper in question by Hollon et al., and many of the papers
referenced in the comment above by Hollon et al. receive research grant monies. These researchers haveacademic positions and have been researching CBT and comparing CBT to medications under
unblinded/no blind placebo control conditions, usually reporting favorable outcomes as referenced in the
Hollon et al. comment (see grant information sections in the Hollon et al. comment references, and9Collorary 5 in Ioannidis ).
In Conclusion
1. There is really no way to get away from the problem in the inability to single- or double-blind, and haveblind placebo control in a CBT clinical trial, whether in the acute, prevention, or long-term treatment phaseof these trials.
2. Hollon et al. wrongly use the term single-blind, and do not seem to provide an understanding of the biasthat can come from the subject-treater system that can not be filtered by blind raters.
3. Problems in doing post hoc analyses, and the comparing of unblinded CBT arms to blinded medicationarms (the “apples to oranges” comparison of arms of completely differing control levels) run frequently inthe trials noted by Hollon et al. who do not provide an understanding of the serious breech of scientificmethodology inherent in these issues.
Hollon et al. also call their post hoc subgroup analysis “exploratory” in the body of their text but yet claim
these subgroups were clearly part of the efficacy of cognitive therapy in their Abstract conclusion.
These points are extremely difficult for us to understand the scientific logic of, much less make strong
Page 12 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

These points are extremely difficult for us to understand the scientific logic of, much less make strong
statements of efficacy in the Abstact and Conclusions of Hollon et al. and related papers referenced intheir comment.
4. A number of papers have noted problems in blinding of CBT trials as well as the specificity of CBT trials,
and directive vs non-directive approaches, referenced in our reply here, and other papers that thiscomment does not have the space to discuss. Our paper here is not the only concern about the validity
and claims of CBT.
We confidently stand by our assertion that the authors of these studies should use extreme caution andstate clearly the problems with making any conclusions that can be drawn from CBT outcome studies thatare unblinded and have no blind placebo control.

References
1. Schulz KF, Grimes DA: Blinding in randomised trials: hiding who got what. Lancet. 2002; 359 (9307):696-700 PubMed Abstract | Publisher Full Text | Reference Source
2. Piantadosi S: Clinical Trials: A Methodologic Perspective, Second Edition. Book. 2005; Wiley Series in
Probability and Statistics (New York). Publisher Full Text | Reference Source
3. Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, Gluud C, Martin RM, Wood AJG, Sterne
JAC: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions
and outcomes: meta-epidemiological study. BMJ. 2008; 336: 601-605 PubMed Abstract | Free Full Text |Publisher Full Text | Reference Source
4. Marchesi C, De Panfilis C, Tonna M, Ossola P: Is placebo useful in the treatment of major depression inclinical practice?. Neuropsychiatr Dis Treat. 2013; 9: 915-920 PubMed Abstract | Free Full Text | Publisher
Full Text | Reference Source
5. Rutherford BR, Mori S, Sneed JR, Pimontel MA, Roose SP: Contribution of spontaneous improvement
to placebo response in depression: a meta-analytic review. J Psychiatr Res. 2012; 46 (6): 697-702
PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
6. Wampold BE, Minami T, Tierney SC, Baskin TW, Bhati KS: The placebo is powerful: estimating placebo
effects in medicine and psychotherapy from randomized clinical trials. J Clin Psychol. 2005; 61 (7):835-854 PubMed Abstract | Publisher Full Text | Reference Source
7. Friedman LM, Furgerg CD, DeMets DL: Fundamentals of Clinical Trials, Third Edition. Springer. 1998.
Reference Source
8. Jaccard J, Becker MA, Wood G: Pairwise multiple comparison procedures: A review. Psychological
Bulletin. 1984; 96 (3): 589-589 Publisher Full Text | Reference Source
9. Ioannidis JPA: Why Most Published Research Findings Are False. PLoS Medicine. 2005; 2 (8): e124
PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
10. Hanrahan C, New JP: Antidepressant Medications: The FDA-Approval Process and the Need for
Updates. Ment Health Clin. 2014; 4 (1): 11-16 Publisher Full Text | Reference Source
11. Klerman GL: The psychiatric patient's right to effective treatment: implications of Osheroff v. Chestnut
Lodge. Am J Psychiatry. 1990; 147 (4): 409-418 PubMed Abstract | Publisher Full Text | Reference Source
12. Fournier JC, DeRubeis RJ, Shelton RC, Hollon SD: Prediction of response to medication and cognitive
therapy in the treatment of moderate to severe depression. J Consult Clin Psychol. 2009; 77 (4): 775-787
PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
13. Lynch D, Laws KR, McKenna PJ: Cognitive behavioural therapy for major psychiatric disorder: does itreally work?A meta-analytical review of well-controlled trials. Psychol Med. 2010; 40 (1): 9-24 PubMed
Abstract | Publisher Full Text | Reference Source
Page 13 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

Abstract | Publisher Full Text | Reference Source
14. Parker G, Fletcher K: Treating depression with the evidence-based psychotherapies: a critique of theevidence. Acta Psychiatr Scand. 2007; 115 (5): 9-24 PubMed Abstract | Publisher Full Text | Reference
Source15. Hollon SD: Cognitive and behavior therapy in the treatment and prevention of depression. Depress
Anxiety. 2011; 28 (4): 263-266 PubMed Abstract | Publisher Full Text | Reference Source
16. Barth J, Munder T, Gerger H, Nüesch E, Trelle S, Znoj H, Jüni P, Cuijpers P: Comparative efficacy ofseven psychotherapeutic interventions for patients with depression: a network meta-analysis. PLoS Med.
2013; 10 (5): e1001454 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
17. Braun SR, Gregor B, Tran US: Comparing bona fide psychotherapies of depression in adults with twometa-analytical approaches. PLoS One. 2013; 8 (6): e68135 PubMed Abstract | Free Full Text | Publisher
Full Text | Reference Source
18. Cuiipers P, Driessen E, Hollon SD: The Efficacy of non-directive supportive therapy for adultdepression: a meta-analysis. Clin Psychol Rev. 2012; 32 (4): 280-291 PubMed Abstract | Publisher Full
Text | Reference Source
Competing Interests: There are no competing interests associated with this comment.

Reader Comment 27 Aug 2015

Steven Hollon, Vanderbilt University, USA
Doubling-blinding is not a requirement for validity claims: A response to Berger. Hollon, S. D., DeRubeis,
R. J., & Lorenzo-Luaces, L
We agree with the authors that it is virtually impossible to conduct a double-blind trial of psychotherapy.
The patient typically knows what he is getting and the treating clinician knows what is being offered. Like
the authors, we have more confidence in inferences drawn from placebo-controlled trials that are
conducted in a double-blind fashion, all else equal.
However, the findings from so-called “single-blind” trials, like the one that we conducted, whereindependent evaluators were kept blind to treatment condition, should not be dismissed when they havepotential clinical implications. It strains the imagination to, for example, attempt to come up with an
explanation as to why expectancy effects should be limited to the third of the sample that was recurrent
and non-chronic, but more severe. In our trial, those patients showed a 30% increment in recovery rates
whereas patients who were chronic or less severe showed virtually no additive effect of additional CBT. 1
These findings need to be replicated before they influence clinical practice, though they echo prior findings
by Thase et al.2 If the findings are replicated, they have the potential to guide improved care for patientswith non-chronic depression, consistent with the goals of personalized care.
We share with the authors a preference for double-blind controls. In an earlier trial we found cognitive
therapy (CT) roughly comparable to medication treatment and both superior to pill-placebo, CT at the levelof a nonsignificant trend.3 We also found evidence that prior CT and continuation medication reduced
rates of relapse relative to medication withdrawal, triple-blinded with respect to medications andsingle-blind with respect to prior CT.4 The advantage for prior CT over prior medication is a relatively
robust phenomenon but one that depends on single-blind comparisons.5 We also found that CT led togreater increments in employment than medication treatment, despite producing comparable reductions in
depression.6 If the authors are correct that cognitive therapy’s effects are a consequence of nonspecificexpectancy effects, it is interesting that those nonspecifics reduce risk for relapse and enhance
employability. Moreover, we found evidence that specific subgroups of patients exist who preferentially
responded to CT vs. medication above and beyond their average improvement (and the converse), a
Page 14 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

responded to CT vs. medication above and beyond their average improvement (and the converse), a
pattern of findings that is difficult to reconcile with the notion that CT is an inert intervention. 7
The authors have a valid point, double-blind controls, when they are actually maintained, increase theconfidence that we can have in the inferences that can be drawn. We disagree with the authors thatinferences cannot be drawn in their absence; it is simply that such a design feature limits the level ofcertainty about the source of the benefit evidenced by patients in the CT condition. Broad findings from themeta-analytic review comparative literature are quite clear; cognitive therapy is about as efficacious asmedications when adequately implemented in the acute phase of treatment and it has an enduring effectthat medications cannot match.8
In this context, findings that CT is superior to non-directive supportive therapies 9,10 suggest theintervention is more than just a placebo. Other directive psychological treatments are also efficacious in
the treatment of depression but the is no debate among researchers that CT is the most widely studied
form of psychotherapy and has the most well-established short and long-term evidence base. Confidence
in these conclusions would be even greater if they had come from studies conducted using double-blind
designs, but this issue echoes across the sciences. Most areas of Astronomy and Evolutionary Biology, forexample, have progressed despite similar limitations. Indeed it would be good if one could conduct a
double-blind study of a psychotherapy, just as it would be good if one could travel back in time to observe
the evolution of the species or back to the moment of the Big Bang to observe the processes that occurred
then. Even in these counterfactual fantasies one still would not be able to use some of the other tools ofscience, such as randomization and experimental manipulation, which would aim to rule out a variety ofalternative accounts.

References
1. Hollon SD, DeRubeis RJ, Fawcett J, Amsterdam JD, Shelton RC, Zajecka J, Young PR, Gallop R: Effectof cognitive therapy with antidepressant medications vs antidepressants alone on the rate of recovery inmajor depressive disorder: A randomized clinical trial. JAMA Psychiatry. 2014; 71 (10): 1157-1164
PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
2. Thase ME, Greenhouse JB, Frank E, Reynolds CF 3rd, Pilkonis PA, Hurley K, Grochocinski V, Kupfer
DJ: Treatment of major depression with psychotherapy or psychotherapy-pharmacotherapy combinations.
Arch Gen Psychiatry.1997; 54 (11): 1009-1015 PubMed Abstract | Publisher Full Text | Reference Source
3. DeRubeis RJ, Hollon SD, Amsterdam JD, Shelton RC, Young PR, Salomon RM, O’Reardon JP, Lovett
ML, Gladis MM, Brown LL, Gallop R: Cognitive therapy vs medications in the treatment of moderate tosevere depression.Arch Gen Psychiatry. 2005; 62 (4): 409-416 PubMed Abstract | Publisher Full Text |Reference Source
4. Hollon SD, DeRubeis RJ, Shelton RC, Amsterdam JD, Salomon RM, O’Reardon JP, Lovett ML, Young
PR, Haman KL, Freeman BB, Gallop R: Prevention of relapse following cognitive therapy versus
medications in moderate to severe depression. Arch Gen Psychiatry. 2005; 62 (4): 417-422 PubMed
Abstract | Publisher Full Text | Reference Source
5. Cuijpers P, Hollon SD, van Straten A, Bockting C, Berking M, Andersson G: Does cognitive behaviour
therapy have an enduring effect that is superior to keeping patients on continuation pharmacotherapy? A
meta-analysis.BMJ Open. 2013; 3 (4): e002542 PubMed Abstract | Free Full Text | Publisher Full Text |Reference Source
6. Fournier JC, DeRubeis RJ, Amsterdam JA, Shelton RC, Hollon SD: Gains in employment status
following antidepressant medication or cognitive therapy for depression.Br J Psychiatry.2015; 206 (4):332-338 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
7. Fournier JC, DeRubeis RJ, Shelton RC, Hollon SD, Amsterdam JD, Gallop R: Prediction of response to
medication and cognitive therapy in the treatment of moderate to severe depression.J Consult Clin
Psychol.2009; 77 (4): 775-787 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
8. Hollon SD: Cognitive and behavior therapy in the treatment and prevention of depression. Depress
Page 15 of 16

F1000Research 2015, 4:639 Last updated: 16 MAY 2019

8. Hollon SD: Cognitive and behavior therapy in the treatment and prevention of depression. Depress
Anxiety.2011; 28 (4): 263-266 PubMed Abstract | Publisher Full Text | Reference Source
9. Barth J, Munder T, Gerger H, Nüesch E, Trelle S, Znoj H, Jüni P, Cuijpers P: Comparative efficacy ofseven psychotherapeutic interventions for patients with depression: a network meta-analysis.PLoS Med.
2013; 10 (5): e1001454 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
10. Braun SR, Gregor B, Tran US: Comparing bona fide psychotherapies of depression in adults with twometa-analytical approaches.PLoS One. 2013; 8 (6): e68135 PubMed Abstract | Free Full Text | Publisher
Full Text | Reference Source
Competing Interests: No competing interests were disclosed.

The benefits of publishing with F1000Research:
Your article is published within days, with no editorial bias
You can publish traditional articles, null/negative results, case reports, data notes and more
The peer review process is transparent and collaborative
Your article is indexed in PubMed after passing peer review
Dedicated customer support at every stage
For pre-submission enquiries, contact research@f1000.com

Page 16 of 16