dataesr/scientific-paragraphs-categorization huggingface dataset:
Diagnostic accuracy of SenseToKnow for autism detection Using all app variables, we trained a model comprised of K = 1,000 tree-based EXtreme Gradient Boosting (XGBoost) algorithms to classify diagnostic groups  26. Figure  2a  displays the area under the curve (AUC) results for the classification of autism versus each of the other groups (neurotypical, nonautism, developmental delay and/or language delay (DD-LD)), including accuracy based on the combination of the app results with the M-CHAT-R/F 2 , which was administered as part of the screening protocol. Based on the Youden Index  27  , an algorithm integrating all app variables showed a high level of accuracy for the classification of autism versus neurotypical development with AUC = 0. 90 (confidence interval (CI) (0. 87-0. 93)), sensitivity 87.
6%, with similar sensitivity levels across sex, race and ethnicity. Diagnostic accuracy for the classification of autism versus nonautism (combining neurotypical and DD-LD groups) was similarly high. The fact that the sensitivity of SenseToKnow for detecting autism did not differ based on the child's sex, race or ethnicity Fig. 1 |  The SenseToKnow app workflow from data collection to fully automatic individualized and interpretable predictions. a, Video and touch data are recorded via the SenseToKnow application, which displays brief movies and a bubble-popping game (see Supplementary Video 1 for short clips of movies and Supplementary Video 2 showing a child playing the game). b, Faces are automatically detected using CVA, and the child's face is identified and validated using sparse semi-automatic human annotations.
Study cohort The study was conducted from December 2018 to March 2020 (Pro00085434). Participants were 475 children, 17-36 months, who were consecutively enrolled at one of four Duke University Health System (DUHS) pediatric primary care clinics during their well-child visit. Inclusion criteria were age 16-38 months, not ill and caregiver's language was English or Spanish. Exclusion criteria were sensory or motor impairment that precluded sitting or viewing the app, unavailable clinical data and child too upset at their well-child visit  29. Table  1  describes sample demographic and clinical characteristics. In total, 754 participants were approached and invited to participate, 214 declined participation and 475 (93% of enrolled participants) completed study measures. All parents or legal guardians provided written informed consent, and the study protocol (Pro00085434) was approved by the DUHS Institutional Review Board.
Diagnosis of ASD required meeting full DSM-5 diagnostic criteria. Diagnosis of DD-LD without autism was defined as failing the M-CHAT-R/F and/or having provider or parent concerns, having been administered the ADOS-2 and Mullen scales and determined by the psychologist not to meet diagnostic criteria for autism, and exhibiting DD-LD based on the Mullen scales (scoring ≥9 points below the mean on at least one Mullen scales subscale; s. d. = 10). In addition, each participant's DUHS EHR was monitored through age 4 years to confirm whether the child subsequently received a diagnosis of either ASD or DD-LD. Following validated methods used in ref. 6, children were classified as autistic or DD-LD based on their EHR record if an International Classification of Diseases, Ninth and Tenth Revisions diagnostic code for ASD or DD-LD (without autism) appeared more than once or was provided by an autism specialty clinic.
Diagnostic group imbalance was addressed by weighting training instances by the imbalance ratio. Details regarding the algorithm and hyperparameters are provided below. The contribution of the app variables to individual predictions was assessed by the SHAP values, computed for each child using all other data to train the model and normalized such that the features' contributions to the individual predictions range from 0 to 1. A quality score was computed based on the amount of available app variables weighted by their predictive power (measured as their relative importance to the model). Performance was evaluated using the ROC AUC, with 95% CIs computed using the Hanley McNeil method  34. Unless otherwise mentioned, sensitivity, specificity, PPV and NPV were defined using the operating point of the ROC that optimized the Youden index, with an equal weight given to sensitivity and specificity  27.
Given that the study sample autism prevalence (π study = 49 328 ≈ 14. 9%) differs from the general population in which the screening tool would be used (π population ≈ 2%), we also report the adjusted PPV and NPV to provide a more accurate estimation of the app performance as a screening tool deployed at scale in practice. Statistics were calculated in Python V. 3. 8. 10, using SciPy low-level functions V. 1. 7. 3, XGBoost and SHAP official implementations V. 1. 5. 2 and V. 0. 40. 0, respectively.
Computation of the prediction confidence score The prediction confidence score was used to compute the model performance and assess the certainty of the diagnostic classification prediction. Given that autism is a heterogeneous condition, it is anticipated that some autistic children will only display a subset of potential autism signs. Similarly, it is anticipated that neurotypical children will sometimes exhibit behaviors typically associated with autism. From a data science perspective, these challenging cases may be represented in ambiguous regions of the app variables space, as their variables might have a mix of autistic and neurotypical-related values. Therefore, the decision boundaries associated with these regions of the variable space may fluctuate when training the algorithm over different splits of the dataset, which we used to reveal the difficult cases.
We counted the proportion of positive and negative predictions of each participant, over the K = 1,000 experiments. The distribution of the averaged prediction for each participant (which we called the prediction confidence score; Extended Data Fig. 1 ) shows participants with consistent neurotypical predictions (prediction confidence score close to 0; at the extreme left of Extended Data Fig. 1 ) and with consistent autistic predictions (prediction confidence score close to 1; at the extreme right of Extended Data Fig. 1 ). The cases in between are considered more difficult because their prediction fluctuated between the two groups over the different training of the algorithm. We considered conclusive the administrations whose predictions were the same in at least 80% of the cases (either positive or negative predictions) and inconclusive otherwise.
Computation of the app administration quality score A quality score is computed for each app administration, based on the amount of available information computed using the app data and weighted by the predictive ability (or variables importance) of each of the app variables. This score, between 0 and 1, quantifies the potential for the collected data on the participant to lead to a meaningful prediction of autism. After we compute for each administration i the confidence score (ρ i k ) k∈ [1,K]   of each app variable (x i k ) k∈ [1,K]   and gain an idea of their expected predictive power (E X [G (X k )]) k∈ [1,K] , the quality score is computed as Quality score (x i ) = K ∑ k=0 E X [G (X k )] ρ i k. When all variables are missing, (ρ i k ) k∈ [1,K]   = (0, … , 0), the score is equal to 0, and when all the app variables are measured with the maximum amount of information, (ρ i k ) k∈ [1,K]   = (1, … , 1), then the quality score is equal to the sum of normalized variables contributions, which is equal to 1.
Geraldine Dawson Statistics were calculated in Python V. 3. 8. 10, using SciPy low-level functions V. 1. 7. 3, XGBoost and SHAP official implementations V. 1. 5. 2 and V. 0. 40. 0, respectively. Custom code that supports the findings of this study is available at the following location: https://github. com/samperochon/Perochon_et_al_Nature_Medicine_2023. As required by the National Institutes of Health, individual-level descriptive data from this study are deposited in the National Institute of Mental Health National Data Archive (NDA) using an NDA Global Unique Identifier (GUID) and made accessible to members of the research community according to provisions defined in the NDA Data Sharing Policy and Duke University Institutional Review Board.
Acknowledgements This project was funded by a  Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) Autism Center of Excellence  Award  P50HD093074  (to G. D. ),  National Institute of Mental Health (NIMH)   R01MH121329  (to G. D. ),  NIMH R01MH120093  (to G. S. and G. D. ) and the  Simons Foundation  (G. S. and G. D. ). Resources were provided by  National Science Foundation (NSF) ,  Office of Naval Research (ONR) ,  National Geospatial-Intelligence Agency (NGA) ,  Army Research Office (ARO) , and gifts were given by Cisco, Google and Amazon. We wish to thank the many caregivers and children for their participation in the study, without whom this research would not have been possible. We gratefully acknowledge the collaboration of the physicians and nurses in  Duke Children's Primary Care  and members of the  NIH Duke Autism Center of Excellence research team , including several clinical research coordinators and specialists.
A portrait is an exercise of impression management: the sitter can choose the impression she or he wants to create in the eyes of others': competence, trustworthiness, dominance, etc. Indirectly, this choice informs us about the qualities that were specifically valued at the time the portrait was created. In a previous paper, we have shown that cues of perceived trustworthiness in portraits increased in time during the modern period in Europe, meaning that people probably granted more importance to be seen as a trustworthy person. Moreover, this increase is correlated to economic development. In this study, we aim to replicate this result, using more controlled databases: 1) a newly created database of European head-of-state sovereigns (N = 966, from 1400 to 2020), that is a database of individuals holding the same social position across time and countries, and 2) a database of very highquality portraits digitized with the same technique, and coming from the same Museum, the Chateau de Versailles database (N = 2,291, from 1483 to 1938).
Using mixed effects linear models, we observed in the first dataset that the modeled perceived facial trustworthiness of these sovereigns' faces increased over time (b = 0. 182 ± 0. 04 s. e. m. , t(201) = 4. 40, p < 0. 001). On the opposite, no effect of time was detected on the portraits of the Chateau de Versailles (b = -0. 02 ± 0. 03 s. e. m. , t(759) = -0. 85, p >. 250). We conclude by discussing the potential of this new technique to uncover long-term behavioral changes in history, as well as its limitations.
These datasets provide valuable quantitative data about the dynamics of past societies, but they are limited in terms of insights into the psychological traits of individuals. On the other hand, historians have analyzed the content of cultural artifacts, be it codes of honor, common and canonical laws, books, portraits, or costumes  [15, 16]. These works have provided important insights about the evolution of mentalities and values. However, they are mostly qualitative, which most often preclude the possibility to test hypotheses about the dynamics and the origins of the changes. Thanks to recent advances in data sciences, it is now possible to move further  [17] [18] [19] [20]. In this perspective, Baumard et al. [21]  relied on the analysis of book descriptions to study the evolution of the prevalence of love stories throughout time and space, showing causal relation between economic development and the rise of love stories.
Classically, first impressions studies rely on participants' subjective evaluations of unknown faces. However, the use of historical material create obstacles to the use of such procedure as participants' evaluations may be influenced by their previous knowledge on the different time period in which the portraits were painted (participants may have a positive or negative a priori on periods that are often described to the general audience as more or less democratic, open or economically favorable). To counter this issue that can artificially induce an increase in the modeled perceived facial trustworthiness with time, we relied on the methods presented in Safra et al. [22]. By relying on an automated extraction of face action units, this method allows for the approximation of human-like evaluations of first impressions that are not influenced by historical cues that are present in the portraits, although it is influenced by the other biases that are present in the participants data on which it has been trained.
The goal of this method is thus to get measures that would approximate what the subjective evaluations trustworthiness that participants would have on these paintings if they were blind to the historical cues present on the portraits (this measure is referred in this paper to as 'perceived facial trustworthiness'). Using large databases (National Portraits gallery, Web Gallery of Arts), Safra et al. [22]  were able to show that cues of perceived trustworthiness in portraits increased during the early modern and modern periods , suggesting that people probably granted more importance to appear as a trustworthy person. Moreover, this increase was correlated to economic development, and predated the democratization of Europe. This last result converges with Modernization theory, as well as with previous works on the long-term evolution of democratic values  [10, 11] , suggesting that the combination of digital humanities and more classical databases used by economists can provide important insights on the evolution of societies.
However, this study suffers from a number of potential bias. In particular, it could be the case that the observed rise of modeled perceived facial trustworthiness is an artifact that results from the sample of the study and the rising proportion of middle-class portraits. If middle class people are more willing to appear trustworthy, the increasing number in the database over time mechanically would lead to an increase of perceived facial trustworthiness. One way to control for this bias is to restrict the analysis to individuals with similar social status. To do so, 1,224 portraits of European sovereigns covering a timespan between the 14 th century and nowadays were collected online. Although political regimes have changed since the Middle Ages, sovereigns, have very similar social status, being by definition at the top of the social hierarchy.
Therefore, any change in the facial representations we could observe could not be attributed to the replacement of a social class by another. Another potential limits of Safra et al. [22]  is that the only study on non-English portraits relied on the use of multiple databases and so included portraits that potentially varied in their resolutions. It is thus important to test whether the association between economic development and the evolution of portraits reported in Safra et al. is also present in high-quality and homogeneous digitized paintings from another country. Therefore, in the present study, we conducted on the Chateau de Versailles portrait database the exact same analyses as Safra et al. , conducted on the National Portrait Gallery  [22]. Following the work of Safra et al. [22] , we focused on facial expressions variations through time.
To do so, a random-forest algorithm was trained on avatars generated via FaceGen, using the model developed by Oosterhof and Todorov  [23]. Our algorithm is based on the activation of the facial muscles (measured by the software OpenFace). This gives the opportunity to free oneself from any intuition or cultural bias related to historical cues, such as clothing, hairstyle, or the background of the portrait  [24]. We also studied the underlying determinant of this increase in pro-sociality. In line with the modernization theory and previous work conducted on contemporary data  [7] [8] [9] ,  Safra et al. (2020)  showed an association between economic resources and modeled perceived facial trustworthiness. We aimed at replicating this effect. We thus collected growth domestic product data from the Maddison Project database  [25]  for each country we had in our dataset.
Material and methods 2. 1. 1. Data collection and pre-processing. Data collection followed an iterative process due to the difficulty of creating a database large enough to carry analyses similar to Safra et al. [22]. In the end, data collection took four steps. All steps were pre-registered on OSF (https:// osf. io/jbs64 and https://osf. io/6hm2c). We originally planned to use only museum websites because we suspected that portraits were of higher quality on these websites (Step 1). We collected the names of sovereigns using Wikidata API, as well as birth and death dates. These lists were drawn for the following countries: England (or United-Kingdom), Denmark, Spain, France, Poland, Portugal, Holy Roman Empire, Sicilia-Sardinia and Sweden. We then entered each name in the search queries of the major art museums.
This first sample was composed of 421 portraits (pre-registered here: https://osf. io/jbs64). The quality checks performed on this sample did not yield significant results: neither the age nor the gender effect was significant (see Statistical Analysis section 2. 1. 3). In line with our pre-registration, the analyses were stopped. The absence of gender effect was probably due to the small size of our sample (N = 421) compared the ones from the National Portraits Gallery (N = 1962) and the Web Gallery of Art (N = 4106) used in Safra et al. [22]. Therefore, we decided to perform a power analysis to evaluate the needed sample size to be able to expect potential significant results. We used the data from Safra et al. [22]  to perform a power analysis (keeping in mind this study used only one database, unlike in this study, and focused on many different social categories, again unlike this study).
164 different sample sizes from N = 100 to N = 1490 were tested. Random sub-samples (1000 for each sample size tested) were drawn from the dataset of the National Portraits Gallery of Safra et al. [22]. In order to have samples similar to the one of sovereigns, portraits dated later than 1950 were excluded from the National Portraits Gallery database before drawing our subsamples. Indeed, there were 202 portraits (10%) dated later than 1950 in the dataset of Safra et al. [22]  compared to 22 (2. 3%) in our dataset. Then we computed a linear model to explain modeled perceived facial trustworthiness with perceived facial dominance, age and gender as control variables and portrait date (time) as variable of interest. Finally, we ploted the p-value of the effect of time in function of the sample size  (Fig 1A).
From this analysis, using the mean p-values and the 95% CI around them, it appears samples bigger than 750 give p-values lower than 0. 05 in more than 96% of the simulations. As we knew that some faces would be excluded because of a bad detection by OpenFace or other exclusion criteria, we decided to collect a total of around 1,200 portraits to secure a sample size of at least 750, and to extend our data collection to new states (Step 2). To get a higher number of portraits, we added several states: Genoa, Medici, Milan, Montferrat, Savoy, Venice, Luxembourg, Belgium, Netherlands, Scotland, Norway, and Russia (Step 2, preregistered). We collected the names of sovereigns using Wikidata API, as well as birth and death dates. We then entered all these names, one by one, in the search queries of the major art museums.
This led us to a total of 643 portraits, which was still under our goal sample size. Therefore, we relaxed our methods and turned to the Wikipedia database (Step 3). The Wikidata portraits of the sovereigns and their spouses were collected as well as their metadata (painting date, painter's name, technique). Finally, if a new portrait was the same as one already present in our database, then it was eliminated (preregistered https://osf. io/ 6hm2c). Again, this method did not achieve the expected number. We thus turned to the entire web (Step 4). We used Google image to look for portraits representing these sovereigns. The same metadata as before were collected. As the number of potential portraits was now much bigger, we randomly selected 100 and then 150 names in our list of sovereigns to look for more portraits (preregistered).
Finally, a total of 581 portraits were collected from steps 3 and 4. For all portraits (regardless of the collection methodology), profile, low quality and undated ones were not collected. The portraits that were not painted during the lifetime of the sovereign were also removed because we assumed that they would not necessarily reflect the image the sovereign would have liked to display. In the end, we obtained 1224 portraits painted between the 14 th and the 20 th century (Fig 1B  ). 2. 1. 2. Computation of perceived trustworthiness and perceived dominance levels. Level of perceived trustworthiness and perceived dominance of faces were computed based on the facial action units automatically extracted by Open Face 3. 1 (https://github. com/ TadasBaltrusaitis/OpenFace) and transformed into perceived facial trustworthiness and perceived facial dominance evaluation scores using a random forest algorithm optimized on the avatars controlled for perceived trustworthiness and perceived dominance generated by FaceGen  [23] , trained using the exact same method and database as the one reported in  Safra et al.
(2020). The OpenFace 3. 1 software is not able to process all types of images. Some pictures were excluded because the software did not detect any face on it (due to basic characteristics such as face orientation or insufficient contrast between the background and the face). Pictures were rated by 3 evaluators (two interns and one the author). Each of the 3 evaluators independently classified the face as accurately detected or not: 0: badly detected; 1: well detected. The three scores were added to attribute a unique detection score to each picture. These detection scores were used as weights for the statistical analyses. Images that were scored three times 0 were therefore excluded from the analyses. Moreover, when all the data from different searches were combined, we were able to identify and eliminate duplicate images.
To test our hypothesis that GDP per capita is the variable responsible for the increase in pro-sociality in Europe, we collected GDP data at the country level from Bolt & van Zanden  [25]. These data go back to the 16th century for some countries. Unfortunately, several kingdoms do not correspond to a current country (e. g. , the Holy Roman Empire) and therefore cannot be assigned GDP data. Eventually, the countries that were assigned GDP data are: UK, France, Italy, Netherlands, Poland, Portugal, Spain, Sweden, Russia, Germany, and Belgium. As GDP data are partial, we assigned to each portrait date the GDP corresponding to the closest available GDP data for its country. However, for 51 portraits, GDP per capita could not be retrieved, analyses including GDP per capita were thus conducted on a subsample of 915 portraits.