dataesr/scientific-paragraphs-categorization huggingface dataset:
g. , structural, functional or effective). The underlying motivation behind MULAN is to use ensemble procedures to provide the best model or description that elude the idiosyncrasies of any particular characterization. In this instance, the characterization of architectures of directed functional connectivity that best explain the data under a known generative process. In this study, we introduce MULAN (MULtiple method ANalysis), a method, which uses an ensemble approach to extract functional connectivity and a statistical evaluation to directly estimate confidence in the final graph. The MULAN ensemble approach combines several methods to calculate functional connectivity between nodes. A genetic algorithm is used to set parameter and threshold values using simulated datasets. A statistical evaluation compares the calculated ensemble connectivity with a large number of simulated datasets to estimate confidence in the final graph.
General strategy When building a functional connectivity graph from experimental time series, it is not possible to verify the validity of the results, since, by definition; the ground truth structure is not available. We propose the general strategy to start with simulated datasets with known ground structures for an a priori selection of existing basic methods (BMs) and Fig. 1. Flowing chart for MULAN strategy. In order to obtain the optimal parameters (step 1) and estimate the confidence in this choice (step 2), we use more than 80 simulated datasets, which are split into 2 groups (one for training and one for validation) with the same generative parameters as for the experimental data to analyze, such as number of nodes and sampling frequency. Step 1: We generate a set of BMs and optimal parameter values using training simulated datasets with different ground truth structures.
Step 2: We validate these choices with the evaluation dataset. Step 3: Once validated we used the chosen BMs and optimal parameters to analyze experimental data. Step 4: We cross-validate the results using sub-datasets. H. E. Wang et al. NeuroImage 166 (2018)  167-184 parameters (step 1, Fig. 1 ). After validation (step 2), the methods and parameters thus selected are used to analyze the experimental dataset (step 3), followed by a cross-validation phase (step 4). Each step uses MULAN ensemble algorithm, which takes the time series as inputs, and which outputs a directed graph. MULAN is an open platform, i. e. it can accommodate any user-designed procedures (e. g. , cross-validation and genetic algorithms) to optimize not only the choice of methods and parameters, but also the strategies used in the MULAN algorithm itself.
Since MULAN ensemble algorithm is used at each of the previously described steps (Fig. 1 ), we start describing the algorithm. Then, we show how to choose the BMs and parameter values (step 1), and how to validate these choices (step 2). Finally, we apply these to experimental electrophysiological datasets (step 3) and show how to cross-validate the results (step 4). We also show how to analyze graphs with a large number of nodes using the human brain as a prototypical large network, the sensitivity of the analysis to artefacts and the computational cost. Finally, we apply MULAN to human fMRI data.
Choice of basic methods Since the ground truth structure is usually not known when analyzing empirical datasets, it is not possible to validate a posteriori the choice of BMs and parameter values. Rather than using a heuristic approach, MULAN strategy is a priori strategy. We simulate the empirical data using the same mechanics, number of nodes, duration, and sampling frequency, but with different ground truth graph architecture to generate multiple datasets for training (Step 1 in Fig. 1 ). MULAN allows modeling any type of activity (EEG, MEG, functional A dataset is analyzed with four basic methods (BM1-BM4), using N sliding windows, thus generating 4ÂN connection matrices. For each BM, we average the N connection matrices to obtain 4 averaged matrices. After bootstrapping and normalization (Methods), each link l I,j between nodes i and j is characterized by four values (between 0 and 1); one value for each BM.
e. j→i. (C) We define five If-Then MULAN rules. For each considered rule, À1 means an unnecessary condition and 1 a necessary one. H. E. Wang et al. NeuroImage 166 (2018)  167-184 MRI etc. ) in any type of network architecture, which can be user-defined. As a prototypical example, let us consider the analysis of a dataset of field EEGs obtained from 30 recording sites. We use a convolution-based neural mass model (NMM) for EEG  (David et al. , 2006; Moran et al. , 2013)  to generate field recordings at the millisecond timescale (Appendix C and Supplementary Fig. 3 ). Brain networks typically reveal a smallworld topology, which is characterized by dense local clustering, while keeping a short path length between any pair of nodes  (Bassett and Bullmore, 2006; Bullmore and Sporns, 2009, 2012; He et al.
, 2007). We consider 50 network architectures composed of subnetworks with local connectivity of variable density (each sub-network has 5 nodes and 5, 10 or 15 links), where links between subnetworks mediate short cuts in the graph (Supplementary Fig. 4A ). In each graph, the connection strengths (CSs) of the links are normalized and can vary between 0 and 1. We obtain 50 different datasets. We first perform a systematic evaluation of 40 BMs  (Wang et al. , 2014)  on the 50 datasets (Supplementary Fig. 4 ). For the sake of simplicity, we consider here two cases in which all CSs have the same value: 0. 8 or 1. We use the AUC value, i. e. the area under the receiver operating characteristic (ROC) curve, to evaluate the performance of each BM (Methods and Supplementary Fig. 5 ). Then, we select 18 different quadruplets of BMs (Supplementary Fig.
For each simulated dataset, the genetic algorithm (Appendix E and Supplementary Fig. 6 ) returns an optimal set of 12 values. For each of the 50 training datasets, we consider that a set of parameters is optimal if, in the MULAN graph (right panel of Fig. 3A ): (i) there are no false positives for CS ¼ 0, (ii) all strong links (CS!0. 8) are identified as such (green), and (iii) if all CSs between 0. 5 and 0. 8 are correctly identified as existing, although they can be labeled as strong (green) or weak (pink). The evaluation pie chart for 50 datasets (Fig. 3B ) indicates that all links are correctly identified for CS!0. 5. Following this training period, we obtain a distribution, S s , of 50 values for each parameter (Fig. 3C ).
Evaluation The training session is designed to obtain the best match between the ground truth structure and the computed graph. To validate the choice of BMs and parameter values, we generate 30 new datasets (step 2 in Fig. 1 ). For each dataset, we randomly pick 20 groups of 12 parameters from S s (Appendix F,G and Supplementary Figs. 7  and 8 ). We thus obtain 20 different graphs for each dataset. We then calculate the median value for each link. In order to assess the validity of the final averaged graph, we need to include two new thresholds (used only for validation purposes, i. e. they are not part of the general algorithm, hence they do not need to be optimized). A strong (green) link is assigned if the median value of the link is larger than θ S and a weak (pink) link for values larger than θ W but smaller than θ S (Appendix H, Fig.
3D ). The evaluation from all 30 datasets (Fig. 3E ) demonstrates the absence of false positives and negatives for CS ¼ 0 and CS!0. 9 respectively. MULAN correctly identifies strong links for CS larger than 0. 8, but below this value false negatives start to appear. The rate of false negatives is 10% for CS ¼ 0. 6, which is acceptable. The number of missed links increases as the connection strength decreases. Importantly, a weak (pink) link is reliably identified as an existing link (with a 0. 0006 false positive rate: 9 links among 12995 links). For comparison, the same evaluation procedure performed on two well performing directed basic methods, PDC and BCorrD, led to much less robust results (Supplementary Fig. 9 ). We conclude that the training procedure allowed finding optimal sets of BMs and parameter values, enabling MULAN to identify the graphs of test datasets.
Analysis of experimental SEEG datasets As a practical illustration of MULAN strategy, we analyzed intracranial stereo-electroencephalography (SEEG) recordings (20 s) from 30 channels obtained from a patient with drug-resistant epilepsy during an idle awake state (Fig. 6A-C  and Supplementary Fig. 11A ). When using BMs, we find that the functional connectivity matrices they return are very different from one another (Supplementary Fig. 11B ), supporting the necessity of an ensemble approach. We follow the strategy described in Fig. 1. In step 1. Instead of using custom-designed small world architectures (designed to evaluate MULAN in the previous section), we now use realistic networks based on human connectomes made of 84 nodes (Appendix K). Although tractography-based connectomes have limitations, they provide the closest estimate of human structural connectivity, which is sufficient for our evaluation (i.
e. it does not affect the generality of our conclusions). We randomly choose 30 nodes out of 84 to avoid overfitting. We use a neuronal mass model generating local field potentials, and we select sampling and dimensionality to match experimental conditions. We down-sample the signals to 250 Hz to reduce computational cost and to avoid having too high autoregressive model orders, and we choose 6 s sliding windows with a 3 s overlap. We select the 4 best performing BMs: BCorrU, COH1, BCorrD and PDC based on their AUC values (Supplementary Fig. 12A ). Applying the genetic algorithm we obtain the distribution of parameter values S 30 e (Supplementary Fig. 12D. For validation (step 2), we use 50 other connectomes from the database (Appendix K) and the parameter set S 30 e determined previously: The evaluation results (Fig.
6D ) for all 50 datasets show that for green strong links output of MULAN there are no false positives for CS ¼ 0 and no false negative for CS > 0:9. In step 3, we use the same group of BMs and parameter values, we analyze SEEG signals and obtain the connectivity graph (Fig. 6E ). In step 4, in order to cross-validate our results, we use the sub-dataset strategy described above (Fig. 5A ). We start with simulated datasets from the first 50 connectomes. From each 30-node dataset, we extract three sub-datasets comprising 20 nodes, each link of the 30 nodes being considered at least once in the 3 sub-datasets. We thus obtain 3 Â 50 training datasets, which allow us to calculate the distribution of optimal parameter values S 20h e (Supplementary Fig. 13A ). We then use these optimal parameters to analyze the remaining 50 simulated datasets for validation.
For each of these 50 datasets, we extract three 20-node subdatasets and obtain 4 MULAN matrices: one for the 30 node time-series and three for the 20 node sub-datasets (Supplementary Figs. 13C-D ). We average each link value to obtain the final MULAN matrix. After averaging, the CSs are distributed between 0 and 1, which allows us to consider several categories of CSs, from weak to strong. We consider five thresholds [0. 1, 0. 3, 0. 4, 0. 5, 0. 7] coded with five colors to obtain the final MULAN graph (Supplementary Fig. 13B ). With the procedure using sub-datasets (i. e. introducing hidden nodes), some of these weaker links are better classified as such (Fig. 6F ). Also this procedure decreases the number of false negatives for links with CSs<0. 9. Therefore, the sub-dataset method can be used to obtain a more precise graph in terms of quantitative connection strengths.
The same strategy is applied to the real SEEG dataset (Supplementary Fig. 13D ) to obtain a new graph (Fig. 6G ). The difference with the first analysis, (Fig. 6E ) is the addition of one link and a finer grain distribution of CSs. The previous results validate our ensemble strategy. We now discuss the computational cost of the algorithm, how to handle datasets with a large number of nodes, and how the presence of experimental artefacts may affect the results. Handling networks with a large number of nodes (application to human connectomes) Human neuroimaging data typically involve large-scale networks. A typical parcellation of the human brain includes 84 nodes  (Fischl et al. , 2004). As ground-truth structures, we use the published connectomes of 100 unrelated subjects (Appendix K). We divide the dataset into two 50 datasets, one for training, one for validation.
When the number of nodes is large (e. g. 84), the training phase could not converge to achieve 0 false links for 50 datasets. However, we always achieve convergence to 0 false Fig. 4. Sensitivity to hidden nodes (A) An example of a designed ground-truth structure is shown in the top middle panel. We remove nodes from the graph (filled blue circle), considering low ( < 3) and high ( ! 3) degrees in the right and left panels, respectively. S1 is the new ground truth structure to be recovered after removing the hidden node and its incident edges. S2 corresponds to S1, with the addition of false positive links (indicated in blue), which may have a common source from hidden nodes (left panel) and an indirect path through hidden nodes (right panel). (B) and (C) show the results obtained for hidden nodes with low and high degrees, respectively.
Left, we show the AUC values for 25 BMs if either S1 or S2 is considered as the ground truth structure. We varied the number of hidden nodes (1, 2, 5, 10, 15 and 20), with Nx indicating that we analyze a network comprised of x nodes (i. e. we removed 30-x nodes, which become hidden nodes). Right, AUC values (top) and COS values (bottom) when using MULAN. Green and blue boxes test the hypothesis that S1 and S2 are the ground truth structures, respectively. Note that MULAN always finds S1, demonstrating that the algorithm is robust to hidden nodes. We used two sets of quadruplet: BCorrU, PCorrU, BCorrD, Hmvar, and BTEU, PTEU, BCorrD, ffDTF and θ1 2 ½0:5; 0:6 and θ2 2 ½0:6; 0:7 with N bt ¼ 20. links when using 30 nodes networks. For each connectome of the training dataset, we use 15 28-node sub-datasets, which is the minimal combination of sub-datasets necessary to account for all links.
For comparison, we report the results obtained from two BMs, PDC (Fig. 7E ) and BCorrD (Fig. 7F ), which belong to the quadruplet of BMs used by MULAN for the ensemble approach. Both BMs fail to recover the structure as compared to MULAN (Top of Fig. 7D-F ). MULAN is robust for a large range of evaluation threshold values, whilst thresholds need to be set in very narrow ranges for the two BMs (Fig. 7E-F ). These results confirm that the ensemble approach of MULAN greatly improves the reliability of the results as compared to the best BMs   6  and Supplementary Fig. 13 ). (b) For example, a dataset with 84 nodes (example in Fig. 7 ) is first divided into fifteen 28-node sub-datasets. We divide them further into 2 sub-groups according to the localization of the channels. By comparing the calculated MULAN graphs on other 50 subjects, we obtain the two optimal parameter sets for the two sub-groups.
For each sub-datasets we obtain a sub-network, which we combine to obtain the final MULAN network. Confidence is assessed using datasets of 50 other subjects (Fig. 7G ). 11B ) and optimal parameter values from 50 training datasets with human connectome ground truth networks (Supplementary Fig. 12D ). (E) Distributions of true/false positive/negative rates from MULAN results as a function of ground truth CS values for 50 validation datasets with human connectome structures. (F) After cross-validation from hidden nodes, the network is refined by introducing 5 CS levels (Supplementary Fig. 13 ) and is evaluated on 50 simulated datasets in (G). and that MULAN is more robust for parameter setting. If we now combine the results of the analysis of the 50 connectomes from the validation dataset, we find no false negatives for CS!0.
Computational cost MULAN computational cost mainly depends on the numbers of sliding windows and the number of nodes. We use 20-s-long datasets and 6 sliding windows. The computational cost of BMs increases exponentially with the number of nodes. However, the computational cost of MULAN algorithm itself is much lower than BMs (Supplementary Fig. 14A ). The cost of 20 runs as a function of nodes is shown in Supplementary Fig. 14B. The computational cost of BMs increases linearly with the numbers of sliding windows, whereas MULAN ensemble algorithm itself is not influenced by the number of sliding windows. MULAN ensemble algorithm works on the mean values of the sliding windows, so that the number of sliding windows does not really increase the computational cost. When using subnetworks for cross-verification, there is a near linear increase of the computational time with the number of nodes (Supplementary Fig.
Artefact sensitivity To assess MULAN sensitivity to artefacts, we use two types of artefacts common to SEEG signals: muscle artefacts and jump artefacts (Fig. 8  A, C ). For each type of artefact (around 1 s long), we systematically add the artefacts to a 30-channel 20-s SEEG dataset. We generate 270 datasets with A t artefacts every 2 s on each A c channels, where A t ranges from 1 to 9 (from the beginning until ends of signals) and A c ranges from 1 to 30. We compare artefact-free and artefact-contaminated results with cosine similarity (Appendix D, Fig. 8B, D ). The algorithm is more robust for muscle artefacts than for jump artefacts. For both types of artefacts, the algorithm is more sensitive on the time axis (number of artefacts) than the space axis (number of channels with artefacts).
Analysis of fMRI data Functional connectivity is commonly used to analyze fMRI datasets, and construct graphs  (David et al. , 2008 ). As a worked example, we use a 6-min fMRI recording in a child with autism spectrum disorder (Di Martino et al. , 2017) (Fig. 9H ). In step 1, we start with 50 simulated fMRI datasets (Appendix C) using 50 different human connectomes as ground truth. We use 180-s sliding windows with a 90-s overlap and select 2 best performing BMs: BCohF, BCorrD from 10 basic methods based on their AUC values (Supplementary Fig. 15A ). Note that, as reported before  (Smith et al. , 2011) , none of the BMs is able to extract the ground truth graph even with an a priori knowledge of the most appropriate parameters. Using the training dataset and 2 BMs instead of 4, we obtain the distribution of parameter values for both dense and sparse sub-networks (Fig.
9A  and B ). Note that fMRI datasets, having very low sampling rates, do not include enough information for extracting the underlying ground truth. We obtain less than 21 false links. In step 2, we evaluate the parameter choice using 50 other connectomes. An example of a connectome from the validation set is shown on Fig. 9C , and the computed MULAN graph is shown on Fig. 9D ). There is a 20% false negative rate for strong links and a 10% false positive rate for CS ¼ 0. MULAN improves the results of individual MBs: BCohF (Fig. 9E ) and (Fig. 9F ). However, the evaluation results for all 50 datasets (Fig. 9G ) shows that, although MULAN does better than individual BMs, it is more difficult to extract ground truth graphs when analyzing fMRI data as compared to electrophysiological signals. Adding the third best BM (BCorrU) on these 50 datasets (Supplementary Fig.
Functional, effective and ensemble connectivity MULAN infers ensemble connectivity by pooling information from various functional connectivity measures. Different functional connectivity methods provide complementary information about statistical dependencies in time-series. MULAN essentially tests the hypothesis that the underlying coupling can be inferred from the information provided by different aspects of functional connectivity. The systematic evaluation of MULAN using thousands of datasets supports our hypothesis. Dynamic causal modeling (DCM)  (Friston et al. , 2003; Friston et al. , 2013)  is the most established approach to estimate effective connectivity (a model of interaction or coupling). DCM is based on a neuronal model that describes causal interactions, which are mediated by unobservable neuronal dynamicsand an observation model that describes the mapping from neural activity to observed responses.
MULAN parameters can be adjusted to different types of data. Importantly, the five fuzzy logic rules used in MULAN depend on the combination of BMs. Although these five rules are robust and widely applicable, users still can add or replace rules to optimize results from their chosen BMs. Because no ground-truth connectivity network is available for empirical datasets, we suggest starting the analysis procedure with a systematic evaluation on simulated data to optimize the BMs and MULAN parameters. This involves generating multiple simulated datasets, with the same number of nodes, signal length and sampling frequency as the empirical dataset. First we perform a systematic evaluation  (Wang et al. , 2014)  to select the best performing basic methods, and their associated parameters. Then we use a genetic method to obtain MULAN parametersas illustrated in this paper.
Hidden nodes The MULAN scheme is robust to the influence of hidden nodes. This result is important, as some network nodes are not observable or measurable in many empirical data (e. g. , deep sources in MEG/EEG). Hence, the presence of hidden nodes or indirect pathways between nodes does not influence the ability of MULAN to detect a link between two observed nodes. We found this behavior particularly useful when using hidden nodes to cross-validate MULAN. Importantly, its low sensitivity to the influence of hidden nodes provides an alternative way to analyze datasets with a very large number of nodes (>50); namely, by simply combining the results obtained from the analysis of sub-networks with a smaller number of nodes. Such a procedure could obtain the satisfied connectivity results by using the subnetwork optimal parameters, because when large networks are characterized optimal parameters are hardly obtained with 0 false links.
Application and limitations of MULAN Although an ensemble approach provides a better estimate of connected graphs as compared to single methods, the accuracy depends upon the individual performance of the BMs. The analysis of fMRI data illustrates this. Even the best performing BMs are not able to extract the correct graph, perhaps because the neuronal responses are convolved with a hemodynamic response function with a very low sample frequency. As a result, MULAN cannot output the correct graph, even if its estimation is better than that provided by individual BMs. In contrast to electrophysiology data for which the convergence criterion can be set to 0 false positive link, in the case of fMRI data, the threshold had to be empirically set to 21 false links for the algorithm to converge. Better performing BMs need to be designed to improve the accuracy of fMRI functional graphs.
Our simulated data corresponds to electrophysiological (MEG or EEG) time series, with inherent spectral structure and correlations induced by (simulated) neuronal processes. For SEEG datasets, we can get confident connectivity results based on the given locations and high time resolutions. Whereas from fMRI time series where the neuronal responses are convolved with a haemodynamic response function and with very low sample frequency, it is more difficult to find the ground-truth graph (because all BMs do not perform well enough). Future work may include other algorithms such as DCM, and then work on the connectivity of hidden states  (Friston et al. , 2003; Friston et al. , 2013). Finally, it is important to note that we assume that the network is in a steady state regime. However, functional links evolve in time, a concept coined as Functional Connectivity Dynamics (FCD)  (Hansen et al.