dataesr/scientific-paragraphs-categorization huggingface dataset:
14/17 However, this is not guaranteed due to the possibility of overfitting in the presence of high-dimensional inputs. Also, expanding the region of fMRI inputs would dramatically raise the computational cost (in time and/or memory) of the training process of regression models. This is why the NSDGeneral ROI appears as an ideal compromise used in most studies (including ours). Using a common ROI also facilitates comparisons between studies. In the future, we may also see more accurate movie reconstruction studies that process temporal patterns together with spatial ones on movie-fMRI datasets  50, 51. Besides improving the reconstruction quality, future work could also design novel experiments and analyses on the NSD dataset using generative models. For instance, in this study, we have shown that we can use generative models to reveal the "optimal" stimulus for anatomically, functionally, or retinotopically-defined ROIs, by analyzing the reconstructions of synthetic fMRI patterns created from the corresponding ROI masks.
Acknowledgements This work was funded by the  Agence Nationale de la Recherche ANR  grants  AI-REPS ANR-18-CE37-0007-01  and  ANITI ANR-19-PI3A-0004. We thank  Yu Takagi  and  Zijin Gu  for sharing their test results with us and also we thank  Alex Nguyen ,  Paul Scotti , and  MedARC  team for helping acquire the replicated results of the Mind-Reader study.
Many technological systems these days interact with their environment with increasingly little human intervention. This situation comes with higher stakes and consequences that society needs to manage. No longer are we dealing with pages: AI systems today may cause serious harm. To address this, we wish to exert a kind of control over these systems, so that they can adhere to our moral beliefs. However, given the plurality of values in our societies, which "oughts" ought these machines to adhere to? In this article, we examine Borda voting as a way to maximize expected choice-worthiness among individuals through di erent possible "implementations" of ethical principles. We use data from the Moral Machine experiment to illustrate the e ectiveness of such a voting system. Although it appears to be e ective on average, the maximization of expected choice-worthiness is heavily dependent on the formulation of principles.
For our domain, we use data from the Moral Machine experiment  (Awad et al. , 2018) , but this is merely intended as a working example, as these data provide people's preferences in an AI-infused domain and are widely available. While the Moral Machine experiment was criticized for its inauthenticity or its lack of relevance to morality, we argue that the experimental results are not to be taken at face value but rather have broader applications, as they may provide an opening for discussion of credences and value-sensitive design processes. We also add to the discussion by introducing what we deem to be the most important and pressing issues for such a solution. In this article, in summary, we review the ethical theories that we formalize, explain how we formalize them, and then present results on the differences between various voting systems.
For AI systems, these rankings are not formulated by an individual but rather by a host of data sourced from different individuals; this means that which option is acceptable most of the time is dependent not on the ethical theories themselves but rather on differences in credences between individuals. To illustrate the effectiveness of Borda voting for agents with multiple stakeholders, we base our argument on an experiment built upon data from the Moral Machine experiment conducted in 2018 by MIT  (Awad et al. , 2018)  as a source of data regarding human preferences. In the Moral Machine experiment, the moral choices of respondents across the world were collected through an online survey. They were required to make decisions in response to variations on the following scenario: an autonomous vehicle begins to have trouble with its brakes just before arriving at a crosswalk.
The moral machine dataset The Moral Machine experiment collected data from 1. 3 million respondents from 233 countries and territories. Although 13 scenarios were presented to each respondent, some of them did not answer them all. In our study, we consider the subset of respondents who provided answers to all 13 dilemmas, corresponding to 51,122 users. Respondents who answered multiple times were also disregarded. The aim of our study was to measure individuals' preferences from an ethical perspective, by drawing up a moral profile for each respondent. As each of them faced 13 different variants of a modern trolley dilemma, we measured their preferences for each of our operationalized theories by determining how many times a principle was either followed or disregarded in their decisions. We therefore analyzed the relevant attributes of each dilemma presented in the experiment.
The right we consider here is the right not to be killed intentionally. Because the trolley dilemma ineluctably deals with the death of at least one being, the autonomous car should not swerve under this principle, because that would be seen as taking the action of killing people on the other side of the road. Contractarian deontology: This principle is based on considering laws as moral rules for taking particular actions, and therefore penalizes illegal behavior. In order to choose which moral principles to focus on, we made some simplifications that are justified in the following section. Some assumptions were also made because not all the criteria in the dataset were relevant or sufficient for a moral principle to fit perfectly. We should also note the glaring lack of virtue ethics, which is a system that is difficult to operationalize well.
In this case, rather than a debate, we take the data already provided by participants and show how they cast their ballots by making a choice in the Moral Machine experiment. A decision in the parliament can be made through different voting schemes. In our study, we consider three such schemes. The majority vote: In a classic majority count, alternatives are voted on by each group of delegates. In each instance, the one that receives more votes wins. Borda count: As defended by MacAskill (  2016 ), the Borda method may appear to be efficient, as it avoids choices that are considered unacceptable. The Borda count employs a ranked voting system, in which points are allocated to each alternative. "My favorite theory": As presented by  Gustafsson and Torpman (2014) , my favorite theory (MFT) can be stated as follows: an option x is only permitted in a situation S for an individual P if said individual P has the highest credence in a particular theory and that theory permits option x in situation S.
In voting, this means that the largest group of delegates always gets to decide the outcome. This theory prescribes the most consistent choices without overly relying on inter-theoretic comparisons. Using the data from the Moral Machine experiment on the choices that individuals made during the experiment, we can construct a hypothetical parliament. This hypothetical parliament can convene and show what outcome they would arrive at, depending on the voting scheme employed. Furthermore, this can tell us something about whether principles are violated or respected.
Ambiguity appears in a proportion of dilemmas where the moral principle in question does not clearly prescribe or forbid the user's decision. Incorporating the situations in which ambiguity appears, we measure what we call a moral profile, which indicates the extent to which a respondent follows each ethical theory. The moral profile of an individual consists of their moral score for each of our ethical theories. The moral score for each theory is defined as the percentage of choices made by the respondent that clearly fit what it promotes. We formalized each of the preference profiles as follows: Sentient utilitarianism: max(lives), no matter what kind. Classical utilitarianism: max(human lives). Hedonistic utilitarianism: max(years lived). The data provide no specific details on the number of years of life saved or lost, only the categories of children, adults, and seniors.
In Borda voting, ambiguity is given a specific role, whereby members of the parliament could award these options 0 points. Overall, due to this subtle change in voting, we expected that more principles would be respected and fewer violations would occur under this scheme. A repository containing all the code can be found here: https://gitlab. com/SietzeKuilman/normative-uncertainty-as-a-voting-problem. Frontiers in Neuroergonomics frontiersin. org. /fnrgo.
This is known as the limits of form problem  (Sepielli, 2010). When we, as developers, model the world for an agent, we necessarily make abstractions and thus leave out (relevant) details. In maximizing expected choice-worthiness, we may give credence to certain possible worlds (e. g. , ones in which animal welfare matters). However, what we take into account and what we do not is roughly determined by our outlook on life and by the extent of our knowledge. In this regard, as a prescriptive moral framework it lacks substance and is difficult to differentiate from My Favorite Theory. If put rubbish in, we get rubbish out. We may be capable of conjuring up possible worlds in which certain theories are true, but which theories we consider in this process is up to us. To hearken back to the previous sections, we included ambiguity as an option for Borda voting because the response always involved a choice between A and B, but that simple choice is also a direct limitation imposed by the dataset: responses were a choice between A and B, while in reality that simply may not be the case.
We leveraged one of the largest deepphenotyped longitudinal autism datasets worldwide (EU-AIMS Longitudinal European Autism Project  [22] ) and our final sample included 333 individuals (161 autistic and 172 neurotypical individuals, age 6-30 years). We collected longitudinal adaptive behavioural (Vineland Behaviour Scale-II, VABS-II) and neuroanatomical (structural magnetic resonance imaging) data at two assessment time points (T1 and T2) separated by ~12-24 months. Following recently published criteria  [23] , we grouped ASD individuals into three clinically meaningful outcome groups -"Increasers", "No-changers", and "Decreasers" in adaptive behaviour (based on VABS-II scores, as in  [16] ). Note that we chose to group individuals based on the VABS-II, because, for the VABS-II (unlike for other metrics, such as the gold-standard Autism Diagnostic Observation Schedule [ADOS] and the Autism Diagnostic Interview-Revised [ADI-R]), there exists an empirical measure of the Minimal Clinically Important Difference (MCID).
Study design Our data was part of the Longitudinal European Autism Project (LEAP) described in  [22]. We included participants if they or their parents/ guardians were able to provide informed written or verbal consent/assent to their participation in this study. Our study was approved by national and local ethics review boards at all study sites and carried out to Good Clinical Practice (ICH GCP) standards. See the Supplementary Material for a full description of clinical assessments, inclusion and exclusion criteria, and ethics review boards.
e. using repeatedmeasures within the same individuals) was unprecedented, given the scarcity of other comparable datasets and the challenges inherent to collecting large-scale longitudinal samples (e. g. cost, logistics, participant drop-out etc. ). Nonetheless, in view of the developmental nature of autism, longer follow-up periods would be desirable to further trace developmental trajectories in this condition. To address this limitation, we are currently collecting additional follow-up data from a third timepoint. Further steps that will move us towards being able to apply our results in the clinic include a replication of our results in an independent sample. The main reason for why we have not yet been able to do this is the specific design of our study (longitudinal collection of multimodal data) and our sample (a heterogeneous group of neurotypical and autistic individuals [men and women] across age, cognitive abilities [e.
g. including intellectual disability], and with a range of co-occurring conditions). Specifically, while the study design and sample represent a strength of our project (as they enabled us to answer a novel question in a uniquely suited dataset), they also prevented us from identifying a comparable dataset to attempt a replication of our findings. We aim to do this once suitable datasets become available. Taken together, these future steps will help consolidate our results in different subgroups along the autism spectrum and thereby establish the context of use in which our results may be applicable (e. g. in children/adults) in the clinic. Combined, such studies will provide a basis for the future development of clinical interventions that target the mechanisms associated with specific (e.
ACKNOWLEDGEMENTS The results leading to this publication have received funding from the  Innovative Medicines Initiative 2 Joint Undertaking  under grant agreement No  777394  for the project AIMS-2-TRIALS. This Joint Undertaking receives support from the  European Union 's  Horizon-2020 research and innovation programme and EFPIA and AUTISM SPEAKS ,  Autistica, SFARI. (The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results. )Any views expressed are those of the author(s) and not necessarily those of the funders ( IHI-JU2 ). This work was further supported by the  European Union Horizon-2020  programme  CANDY  (Grant Agreement No. 847818 ). CMF acknowledges support from the  European Union  and the  German Research Association (DFG).
| Correlations with the neuropsychological functioning Finally, we investigated the relationship between the probability of an avalanche transition matrix and the neuropsychological functioning. At first, we extracted centrality measures for each node in the ATMs, namely, the eigenvector centrality, using the Brain Connectivity Toolbox. 33 Successively, we checked for the normality of data distribution using the Shapiro-Wilk test. We observed that the graph index was not normally distributed in the patients group (p =. 046). For this reason, we applied Spearman's correlation between graph theory index of the statistically significant nodes and the neuropsychological measures. We then applied FDR correction across tests. Considering the relatively small sample size, in order to reduce the effect of the extreme value in the correlation computation, we applied winsorized robust correlations using the WRS2 package in R and Bayesian correlation with the software JASP.
FOR MAREOS DEVELOPERS, we provide two multisite MRI datasets with only simple true effects (i. e. , detectable by most machine-learning algorithms) and two with only simple EoS (i. e. , removable by most MAREoS). First, they should use these datasets to fit machine-learning algorithms after applying the MAREoS. Second, they should use the formulas we provide to calculate the relative accuracy change associated with the MAREoS in each dataset and derive an EoS-removal effectiveness statistic. We also offer similar datasets and formulas for complex true effects and EoS that include first-order interactions. FOR MACHINE-LEARNING RESEARCHERS, we provide an extendable benchmark website to show: a) the types of EoS they should remove for each given machine-learning algorithm and b) the effectiveness of each MAREoS for removing each type of EoS.
, 2007 ) , a batch adjustment method developed for genomics data. Several groups have recently adapted this MAREoS to MRI datasets  ( Fortin et al. , 2018 ; Radua et al. , 2020 ). However, we have shown previously that current MAREoS do not entirely remove all differences between sites. Worryingly, these differences may either inflate or shrink the accuracy. In other words, machinelearning algorithms may either use the remaining EoS "fraudulently ", thus inflating accuracy rates, or fail to detect true effects due to the noise associated with EoS, thus shrinking accuracy rates  ( Solanes et al. , 2021 ). While all MAREoS can remove simple additive differences, we are not aware of a MAREoS able to remove complex EoS, such as discrepancies in covariance (i. e. , the interaction between brain regions).
To avoid reporting biased accuracies, we have provided formulas and an R package to unbiasedly estimate the multisite-corrected accuracy in the presence of residual EoS  ( Solanes et al. , 2021 ). This package may be helpful to ensure that the EoS do not bias the reported accuracy. However, the goal of the community should be to develop a novel MAREoS able to remove complex EoS entirely. Unfortunately, MAREoS developers may face a paradox. To our knowledge, there is no straightforward way to measure the EoS-removal effectiveness. For example, in data with EoS and true effects, a MAREoS may yield higher accuracy than another MAREoS for two opposite reasons. It may either reduce the noise associated with EoS (improving the detection of true effects) or fail to remove the EoS (leading to higher accuracy inflation).
On the other hand, in data with only EoS and no true effects, a MAREoS may yield a lower accuracy than another MAREoS for two opposite reasons again. It may either remove the EoS better (minimizing the accuracy inflation) or degrade the data more (worsening the detection of true effects). To overcome this problem, we designed an approach to objectively measure the removal of EoS and the degradation of the data of a given MAREoS. Furthermore, we also provide: a) datasets to conduct these measurements and b) a benchmark website to allow machine-learning researchers readily know the most appropriate MAREoS depending on the situation.
Methods The strategy presented in this paper builds on the study of the change in accuracy associated with a MAREoS. This accuracy change has opposite meanings depending on whether the dataset has only EoS (i. e. , no true effects) or only true effects (i. e. , no EoS). In datasets with only true effects, an accuracy decrease should only be due to data degradation, a side effect of the MAREoS. Conversely, an accuracy decrease in datasets with only EoS should be due to a correct EoS-removal (plus some potential data degradation). We have noted above that accuracy increases are possible in datasets with both true effects and EoS since the noise associated with EoS may shrink the accuracy  ( Solanes et al. , 2021 ). However, to simplify the following calculations, we tried to avoid datasets mixing true effects and EoS.
We first describe the datasets and the specific machine-learning algorithm that MAREoS developers should apply to achieve that differences between MAREoS depend only on the MAREoS (while not on the datasets or machine-learning algorithms). Afterward, we present the formulas to measure the effectiveness of a MAREoS. Finally, we show an example with the Johnson-Fortin-Radua version of the ComBat MAREoS  ( Fortin et al. , 2018 ; Johnson et al. , 2007 ; Radua et al. , 2020 )  (script available at http://enigma. ini. usc. edu/protocols/statistical-protocols/). Readers only interested in the strategy may directly read the section about measuring the EoS-removal effectiveness of a MAREoS.
Description of the datasets and the machine-learning algorithm Each simulated dataset includes the baseline MRI data (cortical thickness, cortical surface area, or subcortical volumes) from ∼1000 patients from 8 scanner sites, two baseline clinical covariates, and the subsequent responses to a given treatment. The simulated studies would aim to predict the response to the treatment (response vs. no response) from the baseline MRI data ( Figure  1  ). The latter follow normal distributions like those returned by FreeSurfer  ( Radua et al. , 2020 )  and have linear relationships with two simulated clinical covariates. In this section, we first describe the datasets (along with the machine-learning algorithm) to familiarize developers with them. Afterward, we briefly report how we created the MRI data for interested readers.
Two datasets have only simple EoS (i. e. , neither true effects nor complex EoS). The lack of true effects means no relationship between the MRI data and the response. Therefore, machine-learning algorithms should not predict the response. Accuracy should be around 50%, like tossing a coin. However, there are substantial simple differences across sites in response probability and MRI data (e. g. , the cortex is systematically measured thicker in some devices). Most machine-learning algorithms may use these simple EoS to "fraudulently " predict the response, inflating the accuracy. These simple EoS should be removable by most MAREoS. Two other datasets have only simple true effects (i. e. , neither EoS nor complex true effects). Thus, there are significant simple relationships between MRI data and the response to treatment (e.
g. , responders have thicker cortices). Therefore, most machine-learning algorithms should predict the response with > 50% accuracy. To predict the treatment response using the brain imaging data, MAREoS developers should conduct a ten-fold cross-validation using our specific fold distribution. Within each fold, they should fit a lasso algorithm in which the variable to predict is the response to the treatment (coded binarily), and the predictors are the MRI data. For instance, in R, we could use the "glmnet " library  ( Friedman et al. , 2010 ). Developers may download the specific R scripts from https://www. imardgroup. com/mareos-benchmark/ We chose the simple lasso because we assumed it can only detect simple effects. We reasoned that it is a kind of linear model, and linear models cannot detect complex effects (e.
g. , interactions or unknown others) unless they are specifically modeled. We also created datasets with complex true effects or EoS, including first-order interactions between two brain regions or nuclei. To assess the effectiveness of a MAREoS in removing first-order interaction-based complex EoS, we propose using a lasso algorithm with a design matrix that includes the first-order interactions. We chose the lasso with firstorder interactions algorithm because, again, we assumed that, being a We encourage other researchers to describe other complex EoS, create the respective datasets, and add them to the MAREoS benchmark website.
Creation of the datasets For the interested reader, we will briefly report how we created each of these MRI datasets. We first generated normally distributed random data for each FreeSurfer region/nucleus, with means and standard deviations similar to real data  ( Radua et al. , 2020 ). Then, to create simple EoS, we added differences between sites: 𝑦 * 𝑟,𝑖,𝑗 = 𝛿 𝑟,𝑖 ⋅ ( 𝑦 𝑟,𝑖,𝑗 -𝑦 𝑟,𝑖, ⋅ ) + 𝑦 𝑟,𝑖, ⋅ + 𝛾 𝑟,𝑖 where y r,i,j is the cortical thickness, cortical surface area, or subcortical volume of the r th ROI from the j th individual of the i th site, and  r,i and  r,i are the multiplicate and additive EoS of the i th site in the r th ROI. We set both  r,i and  r,i to follow normal distributions across the regions of a site, and  •,i and  •,i to follow normal distributions across the sites.