DOI: 10.5061/dryad.xgxd254dz
title: Supplemental methods for: Circulating plasma proteins differentiate epileptic from psychogenic non-epileptic seizures

description: Objective: To develop a diagnostic test that stratifies epileptic seizure
 (ES) from psychogenic non-epileptic seizure (PNES) by developing a
 multimodal algorithm that integrates plasma concentrations of selected
 immune response associated proteins and patient clinical risk factors for
 seizure. Methods: Daily blood samples were collected from patients
 evaluated in the epilepsy monitoring unit (EMU) within 24 hours after EEG
 confirmed ES or PNES and plasma was isolated. Levels of 51 candidate
 plasma proteins were quantified using an automated, multiplexed, sandwich
 ELISA and then integrated and analyzed using our diagnostic algorithm.
 Results: A 51 protein multiplexed ELISA panel was used to determine the
 plasma concentrations of ES patients, PNES patients, and healthy controls.
 A combination of protein concentrations, TRAIL, ICAM-1, MCP-2 and TNF-R1
 were identified that provided a probability that a patient recently
 experienced a seizure. The diagnostic algorithm yielded an AUC of 0.94 ±
 0.07, sensitivity of 82.6% (95% CI: 62.9-93.0) and specificity of 91.6%
 (95% CI: 74.2-97.7). Further, expanding the diagnostic algorithm to
 include previously identified PNES risk factors enhanced diagnostic
 performance with AUC of 0.97 ± 0.05, sensitivity of 91.3%; (95% CI:
 73.2-97.6), and specificity of 95.8%; (95% CI: 79.8-99.3). Conclusions: We
 have identified four plasma proteins that could provide a rapid,
 cost-effective, and accurate blood-based diagnostic test to confirm recent
 ES or PNES. Classification of Evidence: This study provides Class III
 evidence that variable levels of four plasma proteins, when analyzed by a
 diagnostic algorithm, can distinguish PNES from ES with sensitivity of
 82.6% and specificity of 91.6%.
description: With the overall objective
 of generating a diagnostic algorithm that efficiently stratifies ES from
 PNES, a two-step procedure was used. First, the effect size and a random
 forest approach was used to select a subset of proteins that most
 efficiently stratified cohorts. Second, all possible combinations of the
 selected subset of proteins were used to refine algorithms and the most
 information rich, top performing algorithm was selected. Generating all
 combinations of proteins to include into an algorithm is computationally
 inefficient. To reduce the search space and prevent inclusion of proteins
 with limited information the effect size between cohort means and the
 relative importance of each protein in a bootstrap forest was considered.
 The effect size and 95% confidence interval were
 calculated<sup>18</sup>. The results from the effect size
 analysis was then compared to the results of a Random
 Forest<sup>19</sup> importance analysis, implemented using the
 scikit learn python package. The random forest was generated using 1000
 trials of random sampling with replacement to generate a family of
 classification trees. Gini criterion was used to refine the classification
 trees and the relative importance of each feature was assessed using the
 mean decrease in impurity (e.g. the number of times a feature is used to
 split the tree relative to the number of samples split). To challenge the
 approach and determine the noise floor two fake proteins data sets were
 included. These data set were created using a random number generator that
 selected values from a normal distribution with a mean of 0 and a standard
 deviation of 1.  Proteins with mean concentration
 difference large enough that the confidence interval of the effect size
 did not cross zero and had a relative importance greater than the noise
 floor of the random forest analysis were carried forward to algorithm
 testing. All unique combinations of proteins were generated and used to
 refine a diagnostic algorithm using a logistic regression. Video-EEG event
 diagnosis was used as the gold standard, dependent variable that the
 selected set of protein concentrations was refined against to generate a
 diagnostic algorithm. Each algorithm was assessed using the Akaike
 information criterion (AIC)<sup>20</sup>. Specifically
 algorithms were compared by the number of proteins included in the
 combination and a change of greater than 2 in the AIC
 measure<sup>21</sup>. To further characterize the top
 performing algorithms two cross-validation tests were used, first, a
 leave-one-out strategy was used to assess the algorithms accuracy in
 scoring the excluded samples and second a 10-fold cross validation was
 used to assess the algorithms AUC. The binomial proportional confidence
 intervals for the final algorithm were generated using the Wald test for
 area under the curve and the Wilson score interval for the sensitivity and
 specificity. In
 end, a logistic regression was selected to use as the final classifier
 because it is the industry standard and it generates a probability of
 seizure in addition to a discreet yes/no seizure answer when compared to a
 selected threshold. The logistic regression was compared to multiple
 classification approaches including K Nearest neighbor, Naïve Bayes,
 Decision Tree and a random forest. The scikit learn implementation of each
 algorithm was used in the comparison and default parameters were used with
 exception of using reduced entropy rather than Gini criterion to measure
 the quality of a split in the decision tree and random forest classifiers.
 In both cases using the entropy measure resulted in improved mean accuracy
 and AUC. Algorithm performance was compared using two cross validation
 tests, a leave one out approach was used to generate the mean accuracy and
 a ten-fold cross validation was used to compare AUC. To
 assess the influence of potential confounding parameters (age, gender,
 comorbidities, drug prescription and time elapsed from seizure), the
 correlation with protein concentrations were calculated using the Pearson
 correlation coefficient test. The above parameters were also incorporated
 into an algorithm with the protein concentrations and the AIC was
 evaluated to determine if additional information supported their inclusion
 into the final algorithm. To gain insight into changes
 caused by AED prescription all AEDs were sorted according to the chemical
 class, (Table 2).  The prevalence of each drug was calculated as a sum of
 the number of times a drug from the class was prescribed. Each drug class
 was used to refine an algorithm with the protein concentrations and
 analyzed as above using the coefficient p value and AIC. The carboxamide
 class was further evaluated to identify potential confounding effects on
 protein concentrations by stratifying the ES cohort patients according to
 prescription status and calculating Cohen’s d to quantify differences in
 mean concentrations. The same analysis was also conducted for nonsteroidal
 anti-inflammatory drugs (NSAID), which included the following drugs:
 Diclofenac, Celecoxib, Ketorolac, Aspirin, Naproxen, Ibuprofen and
 Meloxicam. For both AED and NSAIDs the drug list was generated upon
 admission to the EMU, dose and titrations were not considered.
 <strong>AED family analysis
 results.</strong> 
   
 Coefficient p
 value AIC 
 AIC difference
 Proteins   42.27755 -1.55255 Barbiturates 0.80428663 42.74488 -2.01988 Benzodiazepines 0.929771118 39.01684 1.708165 Carboxamides 0.326018539 42.20027 -1.47527 Fatty Acid 0.797100265 41.83963 -1.11463 Fructose derivative 0.825184388 42.7182 -1.9932 Functionalized Amino Acid 0.988352379 42.7237 -1.9987 Gabapentinoid 0.96887308 42.77791 -2.05291 Hydantoins 0.976118306 42.61963 -1.89463 Pyrrolidines 0.877994564 42.21584 -1.49084 Sulfonamides 0.778162766 41.71034 -0.98534