Replication attempt to estimate depression severity by fuzzy logic analysis of emotion‐focused fMRI

Recently, Dr. Lu and coworkers (1) reported the use of a fuzzy logic analysis of emotion-focused functional MRI (fMRI) recordings to derive numeric estimates of depression severity. The potential importance of such findings for the use of MRI in neuropsychiatry induced us to see whether we could replicate them at our site. The regional ethical committee authorized the project. Neuroscientists from the original site came to our site to ensure that identical procedures were used in both studies. The original Chinese version of the 24-item Hamilton Depression Rating Scale was translated and was used for the standardized psychiatric interview of the replication attempt. We tested 33 women, as in the original study. They had responded to newspaper advertisements for participants in a research project on brain imaging and depression. Initial screening by email and by phone indicated that many of the women were clinically depressed. They all satisfied criteria to undergo an MRI scan, had not used a prescribed psychoactive medication for at least 2 months, and provided informed consent to participate in the study after being fully informed on the procedures. fMRI recording took place within 30 min of the psychiatric interview. The fMRI session consisted of a sequence of the emotion-focused images that was identical to that used in the original study. Data analyses were carried out by neuroscientists from the original site. Initially, they were blinded to depression ratings from the psychiatric interview. Under these conditions, the original Sugeno-type fuzzy logic algorithm failed to produce a statistically significant relationship between measures of depression severity provided by psychiatric interview compared with scores derived by fuzzy logic analysis of fMRI recordings (r 1⁄4 0.02; P 1⁄4 0.91; N 1⁄4 33). The failure of the original model parameters motivated efforts to revise the model, and 20 nonblinded datasets were randomly selected for that purpose. A new generic fuzzy logic algorithm was derived to optimize the model with the nonblinded dataset; it produced a statistically significant correlation between Hamilton scores and fMRI-based estimates of depression severity (r 1⁄4 0.59; P 1⁄4 0.006; N 1⁄4 20). However, the newly derived model failed to provide a statistically significant correlation between measures of depression severity when applied to the dataset of the remaining thirteen women (r 1⁄4 0.34; P 1⁄4 0.88; N 1⁄4 13). Then, nonblinded exploratory analyses were carried out in Nanjing, China, in efforts towards further optimization of the model. Under those conditions, inspection of the data led to exclusion of subjects in three categories: (a) healthy subjects with highly dispersed, large activations throughout the brain (two subjects), (b) severely depressed subjects with relatively few brain activations (three subjects), and (c) subjects older than 60 years (four subjects). As a result, a statistically significant correlation (r 1⁄4 0.68; r 1⁄4 0.46; P < 0.005; N 1⁄4 24) was obtained between depression severity as measured by the 24-item Hamilton rating scale and fMRI data assessed by a fuzzy logic algorithm. The authors of this letter are particularly skeptical of findings obtained by nonblind, data-dredging procedures including post hoc exclusion of subjects. Disagreements with colleagues in Nanjing, mainly concerning that topic, have caused us to discontinue the collaboration and to submit the findings here without including them as coauthors. Negative results in replication attempts are, we believe, of particular value in scientific research because they provide an empirical basis by which to evaluate the reliability of findings (2). Here, we report mainly negative results obtained in our attempt to replicate findings reported previously on the use of a fuzzy logic algorithm to estimate the severity of depression, based solely on emotion-focused fMRIrecordings. According to the original report (1), the procedure provides reliable numeric and objective estimates of depression severity that can be applied to classify subjects into depression categories and to track the recovery process. The present findings clearly challenge that viewpoint. First, blinded data analysis by the original procedures failed to provide a reliable relationship between measures of depression severity obtained by the Hamilton depression rating scale and by fuzzy logic analysis. Second, the nonblinded cross-validation procedure also failed to provide a reliable relationship between the two measures of depression severity. Third, even nonblinded, post hoc data analysis with exclusion of subjects, which we view as a questionable procedure, accounted for less than 50% of the variance between the two measures of depression severity, which we view as being insufficient for guiding day-to-day, medical decisions regarding treatment regimens. While the potential of fuzzy logic for analysis of fMRI data in depressive DOI 10.1002/jmri.23833 View this article online at wileyonlinelibrary.com. JOURNAL OF MAGNETIC RESONANCE IMAGING 37:498–499 (2013)