Challenges in the Differential Classification of Individual Diagnoses from Co-Occurring ASD and ADHD Using Survey Data

Authors: aditi  jaiswal
dennis p wall
peter  washington
Venue: IEEE_org_EMBS_BHI_2024_Conference
Published: 2024-07-10 02:33:29

Abstract: Autism and Attention-Deficit Hyperactivity Disorder (ADHD) are two of the most commonly observed neurodevelopmental conditions in childhood. Providing a specific computational assessment to distinguish between the two can prove difficult and time intensive. Given the high prevalence of their co-occurrence, there is a need for scalable and accessible methods for distinguishing co-occurring autism and ADHD from individual diagnoses. The first step is to identify a core set of features that can serve as the basis for behavioral feature extraction. We trained machine learning models on data from the National Survey of Children's Health  to identify behaviors to target as features in automated clinical decision support systems. A model trained on the binary task of distinguishing either developmental delay (autism or ADHD) vs. neither achieved sensitivity >92% and specificity >94%, while a model trained on the 4-way classification task of autism vs. ADHD vs. both vs. none  demonstrated >65% sensitivity and >66% specificity. While the performance of the binary model was respectable, the relatively low performance in the differential classification of autism and ADHD highlights the challenges that persist in achieving specificity within clinical decision support tools for developmental delays. Nevertheless, this study demonstrates the potential of applying behavioral questionnaires not traditionally used for clinical purposes towards supporting digital screening assessments for pediatric developmental delays.

Main paper: 
 i. introduction: 
 Autism is often characterized by the presence of restrictive or repetitive behaviors, asynchronous emotions, sensory sensitivities, and atypical social communication patterns. By contrast, attention deficit hyperactivity disorder (ADHD) is defined by difficulties in attention and impulse control [1] . Although the two conditions differ in their diagnostic descriptions, it is frequently noted that a child with ADHD may exhibit characteristics of autism and vice versa. Prior research studies have investigated the prevalence rates of the co-occurrence of autism and ADHD in both children and adults, suggesting that 20-50% of individuals have a diagnosis of both conditions, and conversely, 30-80% of individuals diagnosed with autism have concurrent ADHD [2] [3] [4] . While the individual diagnosis itself is time intensive, the behavioral overlap between the two conditions further complicates differential classification.Distinguishing between a diagnosis of autism only, ADHD only, and both remains challenging for both clinicians and computers. In recent years, machine learning (ML) has been used to identify salient reduced feature subsets for classifying autism with a minimum amount of information [5] [6] [7] , a more direct approach than other autism diagnostics research leveraging genotypic [8, 9] , phenotypic [10] [11] [12] [13] , and brain imaging data [14, 15] . The results of such feature selection procedures have often been used as the basis for diagnostic crowdsourcing-enabled human-in-the-loop AI workflows [16, 17] , where the outputs of human annotations of the reduced behavioral feature set when observing a home video are fed as input into a classical ML model to perform a final diagnosis [18] . The generalization of human-in-the-loop ML to classify a broader range of pediatric developmental delays has yet to be studied. The first step to enable such workflows for multi-class diagnostics is to identify a salient list of behavioral features that can be analyzed and fed into a classical ML model. Most, though not all, prior ML studies have focused on either a single binary diagnosis, enhancing the individual sensitivity of autism [19] [20] [21] [22] and ADHD [23] [24] [25] [26] classifiers or comparing the two conditions without considering cooccurring diagnoses. A few prior studies have examined the differential diagnoses between the two conditions [5, 14, 27, 28] . These works demonstrate the potential of machine learning to optimize the features for simultaneous diagnoses that could help to build accessible screening platforms. However, these findings focus on gold standard clinical assessments, developed primarily for autism detection, limiting the availability of data related to ADHD.Building upon this rich body of prior work, we aim to identify a subset of behavioral features from the publicly available National Survey of Children's Health (NSCH) data that could potentially be used for differential diagnosis of autism and ADHD, including the possibility of cooccurrence. Using population-based data instead of gold standard clinical assessments not only provides more data samples but also enables the analysis of a potentially more diverse and representative sample of the population. While some prior works have used similar data, they either focused on ADHD classification alone [25] or provided statistical analysis using socio-demographic features [8, [29] [30] [31] rather than building ML models. The objective of our work, by contrast, is to identify the distinct subset of features that can effectively capture the complex behavioral patterns differentiating autism and ADHD co-occurrence between a diagnosis of only a single condition.We used publicly available data from the National Survey of Children's Health (NSCH), a project overseen by the Health Resources and Services Administration Maternal and Child Health Bureau. This survey targets children aged 0 to 17 years and offers comprehensive national and state-level insights into various facets of child health and emotional well-being. Responses to questions on various social determinants of the child are completed by the parents or guardians of the children who live in the same household.We used data from the years 2016 (n=50,212), 2017 (n=30,530), 2018 (n=29,433), 2019 (n=42,777), 2020 (n=50,892), 2021 (n=21,599) and 2022 (n=54,103). For the target variables, we used the survey questions: "Has a doctor or other health care provider EVER told you that the Selected Child (SC) has Autism or Autism Spectrum Disorder (ASD)?" and "Has a doctor or other health care provider EVER told you that SC has Attention Deficit Disorder or Attention-Deficit/Hyperactivity Disorder, that is, ADD or ADHD?" If the answer to either question was yes, we encoded the output as "autism only" or "ADHD only", respectively. If the answers to both the questions were yes or no, they were encoded as "Both autism and ADHD" and "None", respectively. A similar strategy was used to categorize Learning disability, Depression, Anxiety, Behavioral problems, Developmental delay, Speech disorder, Tourette syndrome, and Intellectual disability.The initial analysis of the data revealed that numerous behavioral features contained missing values. This could be attributed to respondents either deeming certain questions inapplicable or choosing not to disclose information, resulting in skipped responses. Records missing any target labels for diagnosis such as Tourette's syndrome, autism, ADHD, anxiety, depression, speech disorder, learning disability, and developmental delay were excluded to ensure the reliability of labels for ML model training. We additionally used sociodemographic variables to ensure balanced representation across different groups.The filtered dataset resulted in 270,978 data points and 365 feature columns. As the study's focus was on identifying behavioral markers for autism and ADHD diagnosis, we discarded all the features not describing any observable behavior. Given the differing proportions of missing values across these features, we partitioned them into two distinct groups (Table I ) and assessed their significance in classification tasks using ML algorithms. Three separate classification tasks were performed on each feature group: (1) a binary classification model for distinguishing individuals with any neurodevelopmental condition from those without any condition, (2) a binary classification model to distinguish individuals with either autism or ADHD from those without any condition, and (3) a multilabel classification model to distinguish autism only, ADHD only, both conditions, and none, generating a probabilistic prediction. For all three models, the dataset was divided into training, validation, and test sets using an 8:1:1 split. Logistic regression (LR) and decision tree (DT) models were used to assess the performance of each classification task independently.The raw dataset is available on the official United States Census Bureau website via www.census.gov/programssurveys/nsch/data/datasets.html. All of our machine learning code is available on GitHub via github.com/ucsfdigitalhealth/NSCH_feature_selection.We combined all features to assess their collective performance and identify which features contribute the most in the predictive tasks. Because some features contained numerous missing values, we applied imputation using a kNN Imputer, leveraging the Euclidean distance matrix to estimate missing values based on the closest data samples. Subsequently, we performed all three classification tasks on this combined feature set. We used the MLxtend sequential forward feature selector (SFS) implementation with a Random Forest (RF) classifier to identify the top 12 features yielding the highest classifier performance as measured by the Area Under the Receiver Operating Characteristic (ROCAUC) curve. Utilizing the reduced feature subset identified, we retrained the model on the original dataset to compare its performance when using all features. The overall workflow of the study is illustrated in Fig. 1 . We conducted a preliminary data analysis on our filtered dataset to understand the sex, race, and age distribution among children with autism, ADHD, and both conditions.The demographic breakdown of the dataset is summarized in Table II . The classification model demonstrates decent performance in correctly distinguishing individuals with any neurodevelopmental condition from individuals with no condition as well as those with autism or ADHD versus neither. However, the performance diminishes for the 4-way classification task as the model tends to erroneously confuse autism with ADHD, and vice versa. Upon closer examination, we found that the probabilities associated with each prediction were often very similar, leading to confusion in the model's predictions (Fig 2 ). Table III presents the evaluation metrics obtained on the test sets for each classification task using individual feature groups and the combined feature set using LR as the better performing classifier. Using forward feature selection on the combined feature set, the top 12 features identified were: age, sex, difficulty in making friends, difficulty in coordinating or moving around, affectionate and tender behavior, recognition of alphabets, easily distracted behavior, ability to play well with others, showing concern when others are hurt or happy, serious difficulty in concentrating or remembering things, tendency to work to finish tasks started, and arguing too much. Interestingly, these same features were consistently highlighted when training models using individual feature groups, further validating the model's performance on the combined feature set. In a series of separate studies [5, 32] using a dataset containing responses from the Social Responsiveness Scale [33], researchers found certain features related to social interaction and attention regulation overlapping with the NSCH dataset, helping to partially validate the contribution of the behavioral features available in the NSCH survey.Our analyses indicate that certain questions related to emotion and social skills contribute significantly to diagnostic uncertainty. Specifically, the feature 'ability to make friends' ranks as the second most important variable with an importance score of approximately 1.4, indicating its strong impact on the model's predictions. In contrast, items such as 'stays calm and in control when challenged' and 'argues too much' show moderate importance, with scores around 0.6 and 0.5, respectively. These differences suggest that while emotional regulation factors contribute to the model, social interaction abilities play a more critical role in diagnostic accuracy. Using the partial dependence plots, we found that the model is more sensitive to changes in memoryrelated conditions when diagnosing ADHD than it is for ASD, making it more prone to diagnostic confusion and potentially leading to over-or under-classification in certain cases. However, despite the adverse impact on model sensitivity, this misclassification underscores the model's capacity to discern patterns indicative of neurodivergence broadly defined.This study emphasizes the importance of establishing behavioral markers that can distinguish between neurodevelopmental conditions with overlapping features by leveraging population-based questionnaire data. Using an optimal subset of behavioral features can aid in preliminary screenings of neurodevelopmental delays. Using the NSCH data, the multilabel classifier demonstrates moderate accuracy in distinguishing between autism and ADHD. It is important to note that although misclassifications may occur, the likelihood of the individuals falling into one of the two categories, if either category is classified, is very high. This aligns with prior work demonstrating that ML has the ability to identify some neurodivergent patterns in NSCH survey data that could aid in screening procedures [5, 32] .There are several limitations of this study that must be noted. Importantly, the majority of the dataset is from White participants, with several groups disproportionately underrepresented. This can lead to potential model biases and thus emphasizes the need for high-quality, clinically annotated datasets with information from diverse populations. Additionally, the NSCH dataset is a survey of various aspects of children's physical, mental, and emotional well-being, with a focus on social and healthcare access factors. This contrasts with dedicated diagnostic tools such as the Autism Diagnostic Observation Schedule (ADOS) [34] and the Autism Diagnostic Interview-Revised (ADI-R) [35] , designed to gauge autism spectrum severity, and the Conners Abbreviated Symptom Questionnaire [36] , aimed at comprehensive ADHD assessment. While some observational behaviors from the NSCH dataset align with existing diagnostic tools, we acknowledge that the lack of external validation of our trained model and limited evaluation of feature selection are limitations of this study. We are actively working towards accessing more comprehensive datasets to further refine our model across different demographic groups and clinically validated features.Additionally, the NSCH dataset contains several missing values. We tried to account for this by reducing the sample size and using model-based imputation techniques. The data loss that arises from fewer samples and discrepancies between the true and imputed values may negatively impact the model's predictive capacity. It is also important to acknowledge potential recall bias inherent in behavioral data obtained from parental questionnaires, and addressing these require datasets that use clinical questionnaires. However, such datasets with corresponding labels of both autism and ADHD are sparse. Finally, it is important to note that co-occurrence is not always diagnosed. A lack of a formal diagnosis does not necessarily mean that the child should not have a diagnosis. Because underdiagnosis of both autism and ADHD is a well-documented phenomenon [37] [38] [39] , it is likely that there were children in the dataset who were reported to only have one diagnosis but should have had both, and some reportedly neurotypical children should have actually had at least one diagnosis.In our future research, we will work on incorporating annotations of behavioral features that can distinguish autism, ADHD, and concurrent autism and ADHD into human-in-the-loop AI models [40] . The features identified in this study will be requested from human annotators, whose ratings will be fed as input into a ML model that will perform the final diagnosis. We ultimately aim to identify core behavioral features for early screening, thus helping to streamline clinical assessments and reduce time-consuming evaluation processes for autism and ADHD. Achieving clinical-grade performance in distinguishing the diagnosis of only autism or ADHD from having a diagnosis of both conditions poses a significant computational challenge. Solving this task is necessary given the high prevalence of concurrent autism and ADHD, and this work provides an initial pass at this problem using parent-reported behaviors.ACKNOWLEDGMENT PW acknowledges funding from the National Institutes of Health (NIH) via the NIH Director's New Innovator Award (award DP2-EB035858). We used the generative artificial intelligence tool ChatGPT by OpenAI only to help with editing the grammar and phrasing of the text, and we further refined its output. 
 
 Appendix:  



Reviews: review: title: title: Review of submission 436
paper_summary: paper_summary: The authors provide their results on applying ML models on a survey dataset completed by the parents with the aim to diagnose Autism Spectrum Disorder (ASD), Attention-Deficit Hyperactivity Disorder (ADHD) or both occurring in the same patient. The data stems from the freely-available National Survey of Children's Health (NSCH) dataset and authors use a standard processing pipeline using logistic regression and decision tree models.
strength_weakness: strength_weakness: Unfortunately, the work several shortcomings. The dataset used is very small and only contains qualitative information from surveys filled by caregivers. The work is completely lacking external validation on another dataset. Moreover, the work is not reproducible as the description is too superficial, for example, not even the programming language or libraries used are given. While the methods used are named no meta-parameters are given as well. This is especially problematic as algorithms such as kNN Impute are used to fill in missing values in the dataset. Without their parameters, other researchers are not able to even come up with the same dataset for their analysis.
questions: questions: I would be interested in the translational aspects of this research. Authors propose achieving clinical-grade performance as one of their future aims. How shall this aim be realized following the presented approach with parent-reported features? How shall features such as "Bounces back quickly when things do not go his or her way", "Argues too much", or "Smiles and laughs a lot" be acquired in a way such that they are comparable between subjects/parents?
title: title: The study examines the challenge of distinguishing ASD from ADHD with machine learning
paper_summary: paper_summary: This document discusses the challenges in distinguishing between Autism Spectrum Disorder (ASD) and Attention-Deficit/Hyperactivity Disorder (ADHD), especially when both conditions co-occur, using survey data and machine learning models. The study highlights that while it is easier to differentiate between those with a developmental condition and those without, distinguishing between ASD and ADHD is more difficult due to overlapping symptoms. Using data from the National Survey of Children's Health, the researchers trained machine learning models to identify key behavioral features that could aid in diagnosis. Although the models performed well in binary classification tasks, they struggled with multi-class classification, often confusing ASD and ADHD. The study also points out the limitations, such as biases in the dataset and missing data, and emphasizes the need for more accurate and comprehensive tools for early screening and diagnosis.
strength_weakness: strength_weakness: 1. Missing values in data were inferred using models, which could be propagating preexisting biases
2. The dataset is predominantly from White participants, leading to potential biases
3. Limited evaluation into the features that could be responsible for the confusion
4. The study relies on parent-reported survey data, which may introduce bias
5. The study does not utilize gold-standard clinical diagnostic tools, like Connor scale and ADOS
questions: questions: 1. What point of the clinical workflow would this tool be utilized?
2. Are there other similar datasets that could be used for external validation and for expanding the demographics tested?
3. Were certain behavioral features more prone to causing confusion between ASD and ADHD?
4. How significant was the impact of missing data on the overall model performance? Did you consider alternative methods for handling missing data, and how might these have affected the results?
5. Could you provide more details on why specific behavioral features were chosen for the final model? How do these features align with or differ from those typically highlighted in clinical assessments of ASD and ADHD?
title: title: Challenges in the Differential Classification of Individual Diagnoses from Co-Occurring ASD and ADHD Using Survey Data,
paper_summary: paper_summary: The paper investigates the challenges in distinguishing between Autism Spectrum Disorder (ASD) and Attention Deficit Hyperactivity Disorder (ADHD), particularly in cases where both conditions co-occur. The authors goal is to use machine learning models to help in the differential diagnosis of these conditions using data from the National Survey of Children’s Health (NSCH) from the years 2016 to 2018.
Various machine learning algorithms are employed, including:

Random Forest
Gradient Boosting Machines
k-Nearest Neighbors
Naive Bayes
Support Vector Machines
 The performance of the models is evaluated using accuracy, sensitivity, specificity, and the Area Under the Receiver Operating Characteristic Curve (ROC AUC).
strength_weakness: strength_weakness: 1. The NSCH dataset may have inherent biases due to its demographic composition and the nature of self-reported survey data
2. The authors should elaborate more about the criteria used for selecting the top 12 features and how these features were determined to be the most predictive.
3. The paper does not address the computational efficiency and scalability of the machine learning models used. Given the large size of the NSCH dataset, it would be beneficial to discuss how the models perform in terms of training time and computational resources, especially if they are to be applied in real-time clinical settings.
4. It would be interesting if possible to compare the performance of the proposed machine learning models with existing diagnostic tools or methods used in clinical practice (if there are any).
questions: questions: No questions