F1000Research 2016, 4:638 Last updated: 04 FEB 2016

CLINICAL PRACTICE ARTICLE

Double-Blinding and Bias in Medication andCognitive-Behavioral Therapy Trials for Major Depressive
Disorder [version 2; referees: 2 approved]
Douglas Berger
Meguro Counseling Center, Tokyo, Japan

v2 First published: 27 Aug 2015, 4:638 (doi: 10.12688/f1000research.6953.1)

Open Peer Review

Latest published: 04 Feb 2016, 4:638 (doi: 10.12688/f1000research.6953.2)

Abstract
While double-blinding is a crucial aspect of study design in an interventional
clinical trial of medication for a disorder with subjective endpoints such as major
depressive disorder, psychotherapy clinical trials, particularly
cognitive-behavioral therapy trials, cannot be double-blinded. This paper
highlights the evidence-based medicine problem of double-blinding in theoutcome research of a psychotherapy and opines that psychotherapy clinical
trials should be called, “partially-controlled clinical data” because they are notdouble-blinded. The implications for practice are, 1. For practitioners to be clear
with patients the level of rigor to which interventions have been studied, 2. For
authors of psychotherapy outcome studies to be clear that the problem in theinability to blind a psychotherapy trial severely restricts the validity of any
conclusions that can be drawn, and 3. To petition National Health Insurance
plans to use caution in approving interventions studied without double-blindedconfirmatory trials as they may lead patients to avoid other treatments shown tobe effective in double-blinded trials.

Referee Status:
Invited Referees

12

reportreportversion 2
published
04 Feb 2016

version 1
published
27 Aug 2015

1 Gordon Parker, University of New South
Wales Australia
2 Keith R. Laws, University of Hertfordshire
UK

Discuss this article
Comments (0)Corresponding author: Douglas Berger (doug@japanpsychiatrist.com)
How to cite this article: Berger D. Double-Blinding and Bias in Medication and Cognitive-Behavioral Therapy Trials for Major
Depressive Disorder [version 2; referees: 2 approved] F1000Research 2016, 4:638 (doi: 10.12688/f1000research.6953.2)
Copyright: © 2016 Berger D. This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which
permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Grant information: The author(s) declared that no grants were involved in supporting this work.
Competing interests: No competing interests were disclosed. The author has no financial interests, activities, relationships, and affiliations otherthan those affiliations listed in the title page of the manuscript. There was no data collected or analyzed for this paper.
First published: 27 Aug 2015, 4:638 (doi: 10.12688/f1000research.6953.1)

F1000Research
Page 1 of 7

F1000Research 2016, 4:638 Last updated: 04 FEB 2016

REVISED

Amendments from Version 1

The sentence, “The unblinded trial makes an unbiased evaluationof the efficacy of these pills impossible.” after Figure 1 wasdeleted because it was left-over from a prior pre-publication draft
and did not fit with the text. Hyphens were inserted into the words“self-esteem” and “make-up” in the text.
See referee reports

Psychotherapy clinical outcome trials for major depressive disorder(MDD) are often described as “randomized”, “controlled”, “singleblind”, etc. These words may not adequately describe the levelof methodologic rigor of the design of a trial for MDD because
the endpoints are subjective symptom ratings1, and the inability to
double-blind MDD psychotherapy outcome trials is a crucial
problem in the methodology of these trials2. Cognitive-behavioral
therapy (CBT) is a widely-used type of psychotherapy in the treatment of MDD, however, CBT is very difficult if not impossible todouble-blind because the subjects are actively involved in thetherapy3. While clinical trials of CBT are often called “single-blind”
because the raters are blind to treatment allocation, “single-blind”
in a clinical trial is actually defined as a case where the subjects areblind, not the raters2,4.
The evaluation of MDD efficacy is more complex in some ways
than that for objective endpoints of, say, tumor size, cholesterollevel, or survival years. MDD may be diagnosed in a variety of persons, some with more psychological distress, and some with moreneurovegetative symptoms. The symptoms of MDD are measured
on rating scales whose scores become the endpoints of the study.
All of the items on these rating scales are subjective, and someitems like hopelessness and low self-esteem are likely to improvewith non-specific aspects of receiving care that include the hopeand expectation inherent in belief in the treatment, compared withother symptoms such as lethargy and insomnia5.
It is well known that non-effective drugs and placebo pills will bothshow an average 30% improvement in depression scores from baseline, not just due to spontaneous improvement from waiting6. This
“placebo effect” is thought to be due to hope and expectation of
improvement on the part of the patient7.
We are also concerned that the term “evidence-based” is used in
descriptions of the validity of a specific therapy without being
clearly defined. While not foolproof, a double-blind design tocontrol for expectations in antidepressant confirmatory studies is
crucial in order to decrease potential bias2,8.
Non-experimental comparative designs may also be used to makeclinical inferences, however, this requires that studies include anumber of conditions including: that the study subjects need toprovide valid observations for the biological question under study;
and the effect of the treatment must be large compared with random error and bias2. These conditions are extremely hard to meet in
MDD where symptom reports are subjective1.
We think that using the term “evidence-based” for both doubleblinded clinical drug trials as well as for unblinded psychotherapy

trials confuses a consistent definition for “evidence-based”. We
opine that the most valid definition of “evidence-based” is that ofevidence garnered from the results of confirmatory trials of antidepressants that require double-blinding (http://cpnp.org/resource/
mhc/2014/01/antidepressant-medications-fda-approval-processand-need-updates)9. In this way, clinical trial designs with thestrongest control level would be the standard for “evidence-based”
data, although we acknowledge that double-blind clinical trials
may also have various design and/or operational problems leading
to invalid results.
We would like to illustrate how the combination of the placebo
effect, along with the inability to double-blind a psychotherapy
trial, can lead to bias in the results.
Figure 1 illustrates the effect hope and expectation vs. pharmacological effect may have on improved depression scores. Ratingsof depressive symptoms are subjective, some symptoms of which
thus may be amenable to a subjective sense of improvement withthe hope and/or expectation of entering a trial. Both subjects and‘treaters’ are blind to the content of the pill received. Group B,
is given a known antidepressant, but the subjects are blind to thenature of the pill. Blinding allows the study to show any unbiased
antidepressant effect additional to hope and expectation.
Even more than just saying a study was “blinded”, however, absolute concealment of what treatment was allocated is crucial inorder to avoid bias10. The study should clearly describe how they
maintained the blind and employ an “exit analysis” to confirm that
subjects were not aware of their treatment allocation.
Figure 2 illustrates a non-blinded psychotherapy efficacy trial,using a “discussion” group and a “CBT” group. Because subjects
are openly receive the intervention given in a psychotherapy, it isessentially impossible to blind a psychotherapeutic intervention.
Any type of psychotherapy could be used in this model, but CBT
is an instructive case because there are workbooks and specifictasks given to subjects making it clear to subject that they are in the
CBT group. The subjects and therapists are both told of the type oftherapy received.
General Discussion refers to a non-directive, non-supportive discussion that is generally assumed to have no effect on MDD andrepresents a group that would have no expectation or hope of receiving a specific and directive therapy like CBT for the purposes ofdemonstrating the effect of unblinding for these groups. Assumingthat “General Discussion” should not be effective in MDD, the discussion group’s improvement would then be similar to the placebo
effect seen in a drug trial, and psychological placebos may also be
as effective as accepted psychotherapies in MDD11.
A third person who rates the degree of depression throughout the
study should not know the therapy received (called a “masked
rater”), but any bias on the part of the subject will just be part of theratings reported by the subject to, and recorded by, the rater. The
study is open so that it is not really possible to assess how muchof each group’s improvement is due to the placebo effect, actual
efficacy, or a bias towards CBT in knowing one is receiving CBT
and/or hope from the act of actively trying to decrease negative
thoughts as is done in CBT.
Page 2 of 7

F1000Research 2016, 4:638 Last updated: 04 FEB 2016

Figure 1. Double-blind randomized drug trial.

Figure 2. Unblinded randomized psychotherapy trial.
Page 3 of 7

F1000Research 2016, 4:638 Last updated: 04 FEB 2016

The importance of blinding in CBT interventions for psychiatricdisorders was supported by a large meta analysis. Controlling forplacebo and blindedness, a meta-analysis of data from publishedtrials of CBT that showed CBT fared no better than non-specificcontrol interventions in the treatment of schizophrenia and did not
improve relapse rates, CBT showed no effect in prevention of bipolar disorder episodes, and only small treatment effects were seen instudies of MDD12.
In our location in Japan, the Japanese National Health Insurance(NHI) system added CBT as a reimbursable procedure for MDD
in 201013. It is concerning to us for a National Health Insurance
system to provide reimbursement for a treatment of MDD that does
not have the same scientific rigor that a double-blind study of anantidepressant would have. This is similar to the NHI situation in
the UK as discussed by Lynch12.
The suicide rate in Japan is among the highest in the world (http://
www8.cao.go.jp/jisatsutaisaku/whitepaper/en/w-2013/pdf/chap11_p2-3.pdf, Accessed on December 4th, 2014.)14, and it is possiblethat treatments for MDD that do not have confirmatory double-blind
clinical trials may lead patients with serious depression away fromother approved treatments that do have this confirmation. The factthat the organizational relationship of the Ministry of Health, Labourand Welfare (MHLW) who funded the CBT studies also decides
on the make-up of the committees at an organization (called the
Chuikyo) that determines approval for reimbursement by the NHI
is also of concern (http://www.japantimes.co.jp/life/2006/03/14/
lifestyle/who-is-paying-the-price-of-health-care/#.VIAGhMlWrkc,
Accessed March 20th, 2015)15.

4. It is imperative that any intervention for a disorder with subjective endpoints such as MDD requires the same rigor in
double-blinding in order to conclude that the results show“efficacy” or are “evidence-based”. This paper proposes to
use the term, “partially-controlled clinical data” in placeof “evidence-based clinical data” for results obtained fromunblinded studies.
The implications for practice are, 1. For practitioners to be clear
with patients the level of rigor to which interventions have beenstudied, 2. For authors of psychotherapy outcome studies to beclear that the problem in the inability to blind a psychotherapy trial
severely restricts the validity of any conclusions that can be drawn,
and 3. To petition National Health Insurance Plans to use caution inapproving interventions studied without double-blinded confirmatory trials as they may lead patients to avoid other treatments shown
to be effective in double-blinded trials.
The limitations of this paper are that the lack of double-blinding
does also not prove that the psychotherapy intervention is not helpful in some way to the indication being treated. Clinical opinion
and consensus may guide how a psychotherapy will be used inpractice.
We hope this paper can stimulate more research related to problems
in blinding of psychotherapy outcome studies, the potential economic and clinical costs of providing or not-providing private or
national health reimbursement for psychotherapeutic interventions,and further discussion on how our official professional organizations and national research centers will define “evidence-based”
in relation to interventions for major depressive disorder.

Conclusions
The conclusions of the rationale presented in this paper would bethat for MDD:
1. Pill placebos show considerable positive effect on disorders with subjective endpoints such as those used to rate
MDD5–7.
2. Psychological placebos may be as effective as accepted
psychotherapies11.
3. Psychotherapy clinical trials are non-blinded studies, andcannot effectively be double-blinded. Calling these studies“single-blind” obfuscates the non-blinded nature of thesestudies and is not in line with the definition of “single-blind”
in a clinical trial4.

Competing interests
No competing interests were disclosed. The author has no financial
interests, activities, relationships, and affiliations other than thoseaffiliations listed in the title page of the manuscript. There was nodata collected or analyzed for this paper.
Grant information
The author(s) declared that no grants were involved in supporting
this work.
Acknowledgements
The author thanks Mr. Burton Berger for help in preparation of the
Figures originally done on Microsoft Excel.

References
1.

Wood L, Egger M, Gluud LL, et al.: Empirical evidence of bias in treatment effectestimates in controlled trials with different interventions and outcomes: metaepidemiological study. BMJ. 2008; 336(7644): 601–605.
PubMed Abstract | Publisher Full Text | Free Full Text

2.

Piantadosi S: Clinical Trials: A Methodologic Perspective, 2nd ed. New York:
Wiley-Interscience; 2005.
Publisher Full Text

3.

4.

5.

Beck AT, Rush AJ, Shaw BF, et al.: Cognitive Therapy of Depression. New York,
NY: Guilford Press; 1979.
Reference SourceFriedman LM, Furgerg CD, DeMets DL: Fundamentals of Clinical Trials, Third
Edition. Springer; 1998.
Reference Source
Cuijpers P, Driessen E, Hollon SD, et al.: The Efficacy of non-directive

Page 4 of 7

F1000Research 2016, 4:638 Last updated: 04 FEB 2016

supportive therapy for adult depression: a meta-analysis. Clin Psychol Rev.
2012; 32(4): 280–291.
PubMed Abstract | Publisher Full Text
6.

7.

Marchesi C, De Panfilis C, Matteo T, et al.: Is placebo useful in the treatment ofmajor depression in clinical practice? Neuropsychiatr Dis Treat. 2013; 9: 915–920.
PubMed Abstract | Publisher Full Text | Free Full Text
Rutherford BR, Mori S, Sneed JR, et al.: Contribution of spontaneous improvement
to placebo response in depression: a meta-analytic review. J Psychiatr Res.
2012; 46(6): 697–702.
PubMed Abstract | Publisher Full Text | Free Full Text

11.

Wampold BE, Minami T, Tierney SC, et al.: The placebo is powerful: estimating
placebo effects in medicine and psychotherapy from randomized clinical
trials. J Clin Psychol. 2005; 61(7): 835–54.
PubMed Abstract | Publisher Full Text

12.

Lynch D, Laws KR, McKenna PJ: Cognitive behavioural therapy for major
psychiatric disorder: does it really work? A meta-analytical review ofwell-controlled trials. Psychol Med. 2010; 40(1): 9–24.
PubMed Abstract | Publisher Full Text

13.

Ono Y, Furukawa TA, Shimizu E, et al.: Current status of research on cognitive
therapy/cognitive behavior therapy in Japan. Psychiatry Clin Neurosci. 2011;
65(2): 121–129.
PubMed Abstract | Publisher Full Text

8.

Naudet F, Millet B, Reymann JM, et al.: Improving study design for antidepressanteffectiveness assessment. Int J Methods Psychiatr Res. 2013; 22(3): 217–31.
PubMed Abstract | Publisher Full Text

14.

9.

Hanrahan C, New JP: Antidepressant Medications: The FDA-Approval Process
and the Need for Updates. Ment Health Clin. 2014; 4(1): 11–16.
Publisher Full Text

Cabinet Office, Government of Japan. White paper on Suicide Prevention in
Japan. 2013.
Reference Source

15.

10.

Schulz KF, Grimes DA: Blinding in randomised trials: hiding who got what.
Lancet. 2002; 359(9307): 696–700.
PubMed Abstract | Publisher Full Text

Otake T: Who is paying the price of health care? Committee slow tobring transparency to matters of national insurance. The Japan Times,
2006.
Reference Source

Page 5 of 7

F1000Research 2016, 4:638 Last updated: 04 FEB 2016

Open Peer Review
Current Referee Status:
Version 1
Referee Report 25 January 2016

doi:10.5256/f1000research.7488.r11959
Keith R. Laws
Department of Psychology, University of Hertfordshire, Hatfield, UK
I have a few comments on Dr Berger's manuscript, which raises interesting discussion points
1. Dr Berger states that “Psychotherapy clinical outcome trials for major depressive disorder (MDD)
are often described as “randomized”, “controlled”, “singleblind”, etc. These words may notadequately describe the level of methodologic rigor of the design of a trial for MDD because theendpoints are subjective symptom ratings” (my italics).
This latter point might benefit from some expanding and clarifying. All current psychiatric ratings
are subjective to some extent (in pharma trials as well), but there are a variety of subjectives. For
example, differences in terms of whether such 'subjective' assessment occurs using a self-rating
tool (like the Beck Depression Inventory: BDI) or a clinician-rating scale (e.g. Hamilton Rating
Scale for Depression). This is especially pertinent in depression trials as self-rating tools like BDI
are frequently used and in effect, this renders trials 'open'.
As Dr Berger states, meta-analyses show quite clearly that open psychotherapy trials can produce
highly inflated effect sizes (see also Jauhar et al. 2014). The latter meta-analysis proves two pointsa) that compared with even single-blind trials, open trials of CBT can inflate effect sizes hugely andb) that contrary to what some say, psychotherapy trials can be very effectively blinded toassessors.
2. I agree that the term ‘evidence-based’ in psychotherapy has been rendered somewhat
meaningless and glosses over -what could be - important variability in terms of what this phrase
covers. In some instances, paradoxically, the phrase is offered as an bland substitute for providing
actual evidence...not unlike the often repeated “as recommended by NICE” - a shorthand way of
avoiding the presentation and evaluation of actual evidence ... as if these phrases were chiselled instone atop Mount Sinai.
3. Study quality is undoubtedly the key factor and conflating evidence from double-blind, single-blindand open trials under the rubric of ‘evidence-based’ is unhelpful - especially to patients who require
guidance on the efficacy of prospective psychotherapy as Dr Berger argues. Indeed, it would bemuch more helpful to patients and clinicians if bodies such as NICE incorporate study quality
directly into their analysis of the evidence-base and hence, their recommendations.
4. Its worth noting in passing that even when meta-analyses might reveal the same numerical effectsize for a drug and a psychological intervention, this does not indicate equivalence of efficacy.
Again, it depends crucially upon the quality of the data underpinning each effect size - so claims
F1000Research
Page 6 of 7

F1000Research 2016, 4:638 Last updated: 04 FEB 2016

4.

size for a drug and a psychological intervention, this does not indicate equivalence of efficacy.
Again, it depends crucially upon the quality of the data underpinning each effect size - so claims
that CBT and anti-depressants have comparable effects really depends on the assumption thatdesigns are equivalent (which they clearly are not). As Dr Berger suggests, drug trials are better
controlled and so, any equivalence is more apparent than real.
5. The proposal to use “partially-controlled clinical data” in place of “evidence-based clinical data”
certainly highlights a key difference, though I guess it all depends upon the extent to which
psychological therapy can be squeezed into the 'RCT template' that has served drug evaluation sowell…RCTs of psych therapies are necessarily at best single blind or at worst (and still very often)‘open’. Some psychotherapy advocates would of course make a special case for psychotherapy that it is simply not amenable to the same paradigms as drug studies and perhaps Dr Berger could
expand on how he views the validity of such counter-arguments.

References
1. Jauhar S, McKenna PJ, Radua J, Fung E, Salvador R, Laws KR: Cognitive-behavioural therapy for thesymptoms of schizophrenia: systematic review and meta-analysis with examination of potential bias.Br J
Psychiatry. 2014; 204 (1): 20-9 PubMed Abstract | Publisher Full Text
I have read this submission. I believe that I have an appropriate level of expertise to confirm that
it is of an acceptable scientific standard.
Competing Interests: No competing interests were disclosed.
Referee Report 23 October 2015

doi:10.5256/f1000research.7488.r10898
Gordon Parker
School of Psychiatry, University of New South Wales, Sydney, NSW, Australia
A straightforward paper, well written and taking the reader through the difficulties of undertaking andinterpreting psychotherapies when no placebo control or equivalent control strategy is available andtherefore worthy of indexing.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that
it is of an acceptable scientific standard.
Competing Interests: No competing interests were disclosed.

F1000Research
Page 7 of 7