dataesr/scientific-paragraphs-categorization huggingface dataset:
22. 3 (McMurdie and Holmes 2013)    following guidelines on online tutorials (http://joey711. github. io/phyloseq/tutorials-index. html), and vegan v2. 5. 2 (Oksanen et al. 2018). The datasets were normalized by rarefaction to their common minimum sequencing depth(COI: 15,575; 18S: 33,916; 16S: 70,474), before analysis of mock communities and environmental samples. To evaluate the functionality of the bioinformatic tools with the mock communities, taxonomically assigned metazoan clusters were considered as derived from one of the ten species used for the mock communities when the assignment delivered the corresponding species, genus, family, or class. Clusters not fitting the expected taxa were labelled as 'Others'. These non-target clusters may originate from contamination by external DNA or from DNA of associated microfauna, or gut content in the case of whole polyps used for cnidarians.
Alpha diversity detected using each pipeline in the environmental samples was evaluated with the number of observed clusters in the rarefied datasets via analyses of variance (ANOVA) on generalized linear models based on quasipoisson distribution models. Homogeneity of multivariate dispersions were verified with the betadisper function of the betapart package v. 1. 5. 1 (Baselga and Orme 2012). The effect of site and LULU curation on community composition was tested by PERMANOVA, using the function adonis2 (vegan), with Jaccard incidence dissimilarities for metazoans and Bray-Curtis dissimilarities for prokaryotes, and significance was evaluated by.
ACKNOWLEDGEMENTS This work is part of the " Pourquoi Pas les Abysses ?" project, funded by  Ifremer  and the eDNAbyss ( AP2016-228 ) project, funded by  France Génomique  ( ANR-10-INBS-09 ) and Genoscope-CEA. We wish to thank  Caroline Dussart  for her contribution to the early developments of bioinformatic scripts,  Patrick Durand  for bioinformatic support, and  Cathy Liautard-Haag  for her help with lab management. We also wish to express our gratitude to the participants and mission chiefs of the  EssNaut16 (Marie-Anne Cambon Bonavita ),  MarMine  ( Eva Ramirez Llodra , project  247626/O30  ( NRC-BI) ,  PEACETIME  ( Cécile Guieu  and  Christian Tamburini) ,  CanHROV (Marie Claire Fabri)  and  MEDWAVES (Covadonga Orejas) cruises. The MEDWAVES cruise was organised in the framework of the  ATLAS  Project, supported by the  European Union   2020 Research and Innovation Programme  under grant agreement number:  678760.
DATA ACCESSIBILITY The raw data of this work can be accessed in the European Nucleotide Archive (ENA) database (project: PRJEB33873), please refer to the supplementary metadata excel sheet for ENA file names. The dataset, including raw sequences, reference databases, and ASV/OTU tables, is accessible via https://doi. org/10. 12770/0b5d250b-8418-4dda-b39c-960c4481df93. Bioinformatic scripts, config files, and R scripts are available on Gitlab (https://gitlab. ifremer. fr/abyss-project/).
frequent) in negative control samples. ASV/OTU tables were refined based on their BLAST or RDP taxonomy. For both assignment methods, clusters unassigned at phylum-level were removed. With BLAST, assigned clusters represented 33% of COI data, 76% of 18S data, and 97% of 16S data. With RDP, assigned clusters represented 95-99% of data. Non-target clusters (i. e. either assigning ASVs using BLAST+ (Basic Local Alignment Search Tool v2. 6. 0) based on minimum similarity and minimum coverage (-perc_identity 70 and -qcov_hsp 80). An initial test implementing BLAST+ to assign taxonomy only to the COI dataset using a 96% percent identity threshold led to the exclusion of the majority of the clusters. Given observed inter-specific mitochondrial DNA Molecular clusters were refined in R v. 3. 5. 1 (R Core Team 2018).
1999), and considering our interest in the identities of multiple, largely unknown taxa in poorly characterized communities, more stringent BLAST thresholds were not implemented at this stage. However, additional filters were performed during downstream processing described below, and only clusters with assignments reliable at phylum-level were retained in the analysis. The Silva132 reference database was used for 16S and 18S SSU rRNA marker genes(Quast et al. 2012), and MIDORI-UNIQUE(Machida et al. 2017)was used for COI. The databases were downloaded from the DADA2 website (https://benjjneb. github. io/dada2/training. html) and from the FROGS website (http://genoweb. toulouse. inra. fr/frogs_databanks/assignation). Finally, to evaluate the effect of swarm clustering, ASV tables were clustered with swarm v2 (Mahé et al.
2015) in FROGS (http://frogs. toulouse. inra. fr/) at d-values (i. e. nucleotide differences) ranging from 1 to 13 (d = 1, 3, 4, 5, 11 for 18S/16S, and d = 1, 5, 6, 7, 13 for COI), based on settings previously used in the literature (Clare et al. 2016; Atienza et al. 2020; Turon et al. 2020; Djurhuus et al. 2017; Cordier et al. 2019; Sawaya et al. 2019; Wood et al. 2019; Laroche et al. 2018; Andújar et al. 2018a). Resulting OTUs were chimera-filtered and taxonomically assigned via RDP and BLAST+ with the databases stated above, using standard FROGS procedures. This article is protected by copyright. All rights reserved prevalent (more renormalization was performed to remove spurious positive results due to reads assigned to the wrong sample (Wangensteen and Turon 2016), the original R script being available at https://github.
com/metabarpark/R_scripts_metabarpark. To test LULU curation (Frøslev et al. 2017), refined 18S and COI ASVs/OTUs were curated with LULU v. 0. 1 following the online tutorial (https://github. com/tobiasgf/lulu). The LULU algorithm detects erroneous clusters by comparing their sequence similarity and co-occurrence rate.
Inter-individual variability in the functional organization of the brain presents a major obstacle to identifying generalizable neural coding principles. Functional alignment-a class of methods that matches subjects' neural signals based on their functional similarity-is a promising strategy for addressing this variability. At present, however, a range of functional alignment methods have been proposed and their relative performance is still unclear. In this work, we benchmark five functional alignment methods for inter-subject decoding on four publicly available datasets. Specifically, we consider piecewise Procrustes, searchlight Procrustes, piecewise Optimal Transport, Shared Response Modelling (SRM), and intra-subject alignment; as well as associated methodological choices such as ROI definition.
For a review of current applications in the literature, see  Haxby et al. 2020. Nonetheless, it remains unclear how researchers should choose among the available functional alignment methods for a given research application. We therefore aimed to benchmark performance of existing functional alignment methods on several publicly accessible fMRI datasets, with the goal of systematically evaluating their usage for a range of research questions. We consider performance to include both (1) improving inter-subject similarity while retaining individual signal structure as well as (2) computational e ciency, as the latter is an important consideration for scientists who may not have access to specialized hardware. Technically up-to-date and e cient implementations to reproduce these results are provided at https://github.
Materials and Methods In this section, we first consider frameworks for aggregating local functional alignment transformations into a single, larger transform (Section 2. 1. 1) that can be applied at a region-of-interest (ROI) or whole-brain scale. We proceed by introducing mathematical notations for functional alignment, then use these notations to discuss the five functional alignment methods included in our benchmark (Section 2. 2). We next describe our procedure to quantify alignment performance using intersubject decoding (Section 2. 3) and a series of experiments aimed at investigating the impact of functional alignment on decoding accuracy (Section 2. 4). Finally, we describe the datasets used to run each experiment (Section 2. 5) and the implementations used (Section 2. 6).
Experimental procedure For each dataset considered (as described in Section 2. 5), we calculated the inter-subject decoding accuracy for standard, anatomical-only alignment and for each of the five considered functional alignment methods. To calculate inter-subject decoding accuracy, we took the trial-or condition-specific beta maps generated for each dataset (see Section 2. 5 for full details on betamap generation) and fit a linear Support Vector Machine (SVM). In order to ensure fair comparisons of decoding accuracy across experiments, we chose a classifier with no feature selection and default model regularization. Classifiers were implemented in scikit-learn  (Pedregosa et al. 2011) , and decoding accuracy was assessed using a leaveone-subject-out cross-validation scheme. That is, the linear SVM was trained to classify condition labels on allbut-one subject and the resulting trained classifier was used without retraining on the held-out subject, providing an accuracy score for that cross-validation fold.
For each dataset, we first calculated the inter-subject decoding accuracy using standard anatomical alignment. This served as a baseline accuracy against which we could compare each functional alignment method. Using alignment data, functional alignment transformations were then learnt for each pairwise method, where the leftout subject for that cross-validation fold was the target subject for functional alignment. Inter-subject decoding accuracy was then re-calculated after applying functional alignment transformations to the decoding beta maps. In the special case of SRM-which calculates an alignment from all provided subjects in a single decomposition-we withheld the left-out subject from the shared response estimation step to avoid data leakage and finally learned its projection to the resulting shared response space, in which decoding was performed.
Main experiments Experiment 1 uses the experimental procedure described previously to assess accuracy gains provided by alignment methods with respect to anatomical alignment when applied on whole-brain images. We benchmarked the five methods described in Section 2. 2: piecewise Procrustes, searchlight Procrustes, piecewise Optimal Transport, SRM, and intra-subject alignment, with relevant hyperparameters selected as described previously. Results of this benchmark (on five tasks from four datasets as described in Section 2. 5) are presented in Section 3. 1. For each method, we also assessed its computation time relative to piecewise Procrustes alignment. Piecewise Procrustes provides a reasonable computational baseline as it is the only considered alignment method that does not include a hyperparameter and therefore shows a stable computation time across experiments.
5 for details for each dataset). Results from Experiment 2 are shown in Section 3. 2. Experiment 3 tackles the notoriously hard problem of understanding how each of the considered methods align subjects by examining qualitatively their impact on activity patterns across individuals. To "open the blackbox," we reused IBC dataset full-brain alignments learnt in Experiment 1. Specifically, we consider the transformation to sub-04's activity pattern from all other subjects's functional data. With these transformations, we align two contrasts from the IBC dataset: Rapid Serial Visual Presentation of words (RSVP language task) and sound listening. Finally, we run a group conjunction analysis  (Heller et al. 2007 ) on these aligned contrasts and display its results. This statistical analysis, more sensitive than its random e↵ect equivalent on small samples, allows one to infer that every subject activated in the region with a proportion showing the e↵ect consid-ered.
Control analyses In addition to our three main experiments, we ran three additional control analyses on the IBC dataset. First, we aimed to assess the impact of the brain parcellation and its resolution on piecewise alignment by comparing whole-brain decoding accuracy for two IBC dataset tasks using piecewise Procrustes across both data-driven and pre-defined parcellations (Section S4). Second, we calculated inter-subject decoding performance after applying Gaussian smoothing kernels of several widths on both IBC dataset decoding tasks (Section S5). Gaussian smoothing is of particular interest as a comparison to functional alignment, as it is commonly used to facilitate inter-subject comparisons by smoothing over residual variance in functional mappings. Finally, in a third control experiment, we assessed the impact of whether data is represented on the surface or the volume and resolution on decoding accuracy in the IBC RSVP language task (Section S6).
The four datasets used in this benchmark, where each dataset consists of S subjects. We note the alignment data used for each dataset, showing the range of possible task structures which work for alignment-from static images for BOLD5000, to statistical contrast maps for IBC, to complex audio or audiovisual movies for Forrest and Courtois Neuromod. We also note the decoding task(s) used for each dataset. Of note, IBC dataset has two independent decoding tasks, bringing the total number of decoding tasks to five. Each subject's decoding task data comprises d images evenly divided across the listed stimulus categories (except for BOLD5000 categories which are unbalanced).
Datasets and preprocessing In order to assess the performance of each functional alignment method in a range of applications, we searched for publicly accessible datasets that included both a task suitable to learn the alignment (e. g. naturalistic or localizer protocols) as well as an independent decoding task on which we could evaluate functional alignment performance. After discarding datasets where we could not obtain above-chance accuracy levels for within-subject decoding, we retained four datasets: BOLD5000  (Chang et al. 2019) , Courtois-NeuroMod  (Boyle et al. 2020 ), Individual Brain Charting (IBC;  Pinho et al. 2018) , and Study Forrest  (Hanke et al. 2016). For the IBC dataset, we included both a language (RSVP language) and auditory (Sounds dataset) decoding task, yielding a total of five decoding tasks that probe visual, auditory and language systems.
For a complete description of the alignment and decoding data included in each experiment, please see Table  I. BOLD5000, StudyForrest and Courtois-NeuroMod were preprocessed with fMRIPrep  (Esteban et al. 2019) , while IBC data were preprocessed using an SPM-based pipeline as described in  Pinho et al. 2018. A complete description of the fMRIPrep preprocessing procedures is available in the appendix (Section S1). Preprocessed data were then masked using a grey matter mask, detrended, and standardized using Nilearn  (Abraham et al. 2014 ). To reduce the computational cost of functional alignment, we downsampled all included datasets to 3mm resolution. Decoding task data were additionally smoothed with a 5mm Gaussian kernel. A general linear model (GLM) was fit to each decoding task run to derive trial-specific beta maps (or condition-specific beta maps for the Courtois Neuromod and IBC Sounds tasks), which were carried forward for inter-subject decoding.
Implementation With the exception of Courtois Neuromod, all other included datasets are available on OpenNeuro  (Poldrack et al. 2013 ) under the following identifiers: ds000113 (Study Forrest), ds001499 (BOLD5000), and ds002685 (IBC). Courtois Neuromod 2020-alpha2 release will be available under a data usage agreement as outlined on https://docs. cneuromod. ca. Our pipeline entirely relies on open-source Python software, particularly the SciPy stack  (Virtanen et al. 2020). All included methods are implemented in fmralign or accessed through their original, open source implementations as described in Section 2. 2. To ease replication and extension of the presented results, we have created the fmralign-benchmark repository under https://github. com/neurodatascience/fmralign-benchmark. This repository provides an implementation of the procedures adopted in these experiments, building on fmralign and previously cited tools.
FIG. 5. Decoding accuracy improvement and computation time after whole-brain functional alignment. In the left panel, we show decoding accuracy improvement for each of the considered functional alignment methods at the whole-brain level of analysis. Each dot represents a single subject, and subjects are colored according to their decoding task. To aggregate results across datasets, we show accuracy scores after subtracting inter-subject decoding accuracy for the same leave-onesubject-out cross-validation fold with anatomical-only alignment. In the right panel, we show the computational time for each of the considered methods. All computation times are depicted as relative to piecewise Procrustes. For both panels, each box plot describes the distribution of values across datasets, where the green line indicates the median.
Alignment substantially improves inter-subject decoding accuracy Overall, we can conclude that most functional alignment methods consistently improve decoding accuracy with gains from 2-5% over baseline. This trend is relatively consistent across datasets and target subjects. Thus, alignment methods manage to reliably reduce individual signal variability while preserving task-relevant information in a variety of conditions. Although betweendataset variance yields large boxplot, these methods have significant e↵ect. Indeed, as reported in Table  S1 , baseline accuracy is around 20% above chance across datasets. In this setting, a 5% average improvement across datasets is a substantial increase. In order to further contextualize these results, we also estimated the noise ceiling for inter-subject decoding.
SRM shows higher performance in ROI-based decoding Uniquely, the Shared Response Model (SRM) shows decreased whole-brain decoding accuracy compared to ROI-based decoding accuracy. Specifically, as visible in Figure  7 , SRM yields the best accuracy improvement at the ROI level of analysis at a low computational cost (five times quicker than piecewise Procrustes). In ROI-based setting, the baseline is around 10 % above chance in all datasets (except Neuromod). Hence SRM 3% accuracy increase is a strong e↵ect(see Table  S2 ). SRM is not well-suited to full-brain decoding because it estimates the shared response using a limited number of components. Projection into a common space results in sharp data rank reduction and thus a loss of information compared to using original data. In an ROI, by contrast, we have a smaller number of voxels p compared to the number of training samples n.
All displayed maps were thresholded at 1/3 of their maximum value. We see that functional alignment yields stronger contrasts overall when compared to anatomical alignment. Piecewise Procrustes and piecewise Optimal Transport yield less smooth representations, better preserving signal specificity. alignment using inter-subject decoding, and we have used this framework to benchmark five functional alignment methods on five distinct decoding tasks across four publicly available datasets. In general, we find that functional alignment improves inter-subject decoding accuracy in both whole-brain and ROI settings. These results, combined with our quali-tative visualization of the e↵ects of functional alignment on signal structure, suggest that functional alignment improves inter-subject correspondence while matching signal to realistic functional topographies.
This extend conclusions from earlier work  (Güçlü and van Gerven 2015, Guntupalli et al. 2016). Specifically, the baseline intersubject decoding accuracy is roughly 20% above chance across datasets (Table  S1 ). The best performing method in our benchmark, piecewise Optimal Transport bring a substantial 4% improvement over this baseline on average, recovering roughly half of the accuracy lost to intersubject variability. Although we see improved decoding accuracy on average, we also note that the functional alignment methods considered show di↵erent performance at the ROI and whole-brain scale. While piecewise Optimal Transport performs best when aggregating transformations across the whole-brain, SRM performs best to enhance decoding within regions of interest, bringing a 3% important improvement over a baseline only 10-15% above chance (Table  S2 ).
Combining local alignment models Across datasets, we find that the spatial framework of alignment and decoding significantly a↵ects subsequent performance. Notably, piecewise Procrustes outperforms searchlight Procrustes, both in terms of accuracy and computational performance. The methodological di↵erence between these methods is whether alignment transformations are learnt within overlapping neighborhoods (as in searchlight Procrustes) or not (as in piecewise Procrustes). Searchlight alignment su↵ers in that the overlap between searchlights requires multiple computations for a given neighborhood, and the aggregated transformation is no longer guaranteed to reflect properties of the original transforms, e. g. orthogonality. Although piecewise alignment may introduce discontinuities at parcel boundaries, in our results this overall benefited decoding performance.
Evaluating alignment performance with decoding We use inter-subject decoding to quantify the amount of mutual information recovered by functional alignment methods. In general, identifying publicly available datasets with tasks appropriate for both inter-subject decoding as well as functional alignment remains a challenge. Beyond the four datasets included in these results, we investigated several other publicly available datasets such as the Neuroimaging Analysis We had di culties in achieving su cient baseline accuracy levels in these and other datasets, and we therefore chose not to include them in the present study. This suggests that the amount of signal discriminating complex experimental conditions is not strong enough to find inter-subject patterns robust to variability in many publicly available datasets, likely due to limited sample sizes and inappropriate experimental designs.
Study limitations and future directions Although our study provides a broad evaluation of the performance of several functional alignment methods, there are several dimensions which we hope future work will better address. Notably, we did not thoroughly investigate how alignment performance is impacted by image resolution and whether data are represented on the surface or the volume. Using volumetric images downsampled to a standard resolution of 3mm isotropic enabled us to make fairer comparisons across datasets at a reasonable computational cost. We also show in Section S6 that results from piecewise Procrustes alignment on the IBC dataset hold in a higher resolution, surfacebased setting. Nonetheless, other functional alignment methods might show di↵erent patterns of performance in this setting or at di↵erent resolution levels.
Conclusion In the present work, we have provided an extensive benchmark of five popular functional alignment methods across five unique experimental tasks from four publicly available datasets. Assessing each method in an intersubject decoding framework, we show that Shared Response Modelling (SRM) performs best at a region-ofinterest level of analysis, while Optimal Transport outperforms other methods at the whole-brain scale. Our results support previous work proposing functional alignment to improve across-subject comparisons, while providing nuance that some alignment methods may be most appropriate for a given research question. We further suggest that identified improvements in inter-subject decoding demonstrate the potential of functional alignment to identify generalizable neural coding principles across subjects.