dataesr/scientific-paragraphs-categorization huggingface dataset:
Hyperparameter optimization To select the best hyperparameters for each dataset and each type of initialization, we evaluated the performance of each model by performing a 5-fold cross-validation on the validation dataset. For each type of classifier (i. e. , initialized with default algorithm vs. pretrained), we refined and optimized the hyperparameters using the largest datasets (Large BrainPedia and HCP). However, the large amount of training data made it computationally extremely costly to perform a full grid search. We therefore limited our research to predefined values of batch sizes (32 or 64), number of epochs (200 or 500), and model architectures (4 layers or 5 layers). All batch sizes, number of epochs, and architectures were tested for each type of classifier and each dataset. We did not perform any optimization on the learning rate to limit the computational cost of our experiments.
Homogeneous dataset (single study) The HCP dataset was used to compare the performance of the models for the task of decoding on a homogeneous dataset (i. e. , from a single study). We studied the impact of 2 factors on the classification: sample size and number of target classes. For sample size, subsets of the global HCP dataset were created with different number of subjects: N = 50, 100, and 200, with each smaller subset being a subset of the immediately larger one. To create these subsets, we first split the global HCP test dataset into 5 folds, with different subjects in each fold. In each of these 5 folds, we randomly sampled 200/5 = 40 subjects and obtained 5 subfolds that together composed the smaller subset of 200 subjects. This process was repeated for subsamples N = 100 and 50 by sampling from their superset.
This ensured that the 5 models trained on different combinations of the 4 folds of a smaller subset could be tested on the remaining fold of the global test dataset with no overlap between the training and test data. The process is illustrated in Fig. 3A. In the end, we obtained 4 datasets with respectively N = 50, 100, and 200 subjects in addition to the global dataset with all subjects (N = 393). These datasets respectively contained 1,150, 2,300, 4,590, and 9,017 statistic maps in the test subset and 1,150, 2,300, 4,591, and 9,053 in the validation subset (note: some contrasts were missing for part of subjects). Since we use a 5-fold validation scheme, the models were trained on approximately 80% of the statistic maps in the corresponding subset (i. e. , validation for hyperparameter optimization and test for performance evaluation).
Three types of classification were investigated: first, the "contrast classification," which consisted in identifying the contrast associated with a statistic map (23 different contrasts); second, the "task classification," which consisted in identifying the task associated with a statistic map (7 different tasks, with multiple contrasts per task); and third, the "1-contrast task classification. " This time, we selected a single contrast per task and classified the tasks (7 different tasks, with 1 contrast per task). The selected contrasts were "2-back places," "faces," "punish," "relational," "right hand," "story," and "tom," respectively for the tasks "working memory," "emotion," "gambling," "relational," "motor," "language," and "social. " We selected these contrasts similarly to what was done in  [8] , in which the HCP dataset was used in a decoding model.
For each task, the contrast that showed a greater association with the task had priority over the other (e. g. , "punish" for the "gambling" task). For "working memory" and "motor" tasks, which contained more than 1 task condition, the authors of  [8]  randomly chose one ("2back body" for working memory and "right hand" for motor). The dataset used for this third type of classification was thus smaller than the others (only 1 map per task per subject). For this classification task, the number of statistic maps was respectively 300, 598, 1,198, and 2,355 for N = 50, 100, and 200 and for the global dataset.
Heterogeneous dataset (multiple studies) To study the benefits of self-taught learning on a heterogeneous dataset (i. e. , from multiple studies), we used BrainPedia. For these experiments, we focused on the classification of mental concepts (as available in NeuroVault metadata). Fig. 3B  illustrates the process used to split this dataset. To perform the split while maintaining the heterogeneity in each fold, we randomly sampled 50% of the subjects of each study to form the "validation" and "test" datasets of BrainPedia (see Fig. 3B ). Then, each dataset, each study was split into 5 folds, and the nth folds of the different studies were combined to form the nth fold of the dataset. Validation and test datasets included N = 428 subjects and were respectively composed of 3,179 and 3,269 statistic maps.
We also studied the impact of sample size in the presence of heterogeneity by extracting smaller datasets. Among the 29 studies of the BrainPedia dataset, we only kept those that were composed of more than 20 subjects. In these remaining studies, already split into 5 folds in BrainPedia validation and test subdatasets, 2 subjects were randomly drawn per fold per study per subdataset to obtain 10 subjects per study per subdataset. Like above, the nth folds of the different studies were combined to form the nth fold of each subdataset of the "Small BrainPedia" dataset. In the end, this smaller dataset was composed of 1,844 maps, divided in 30 classes, from 11 studies and 220 subjects. This dataset was also split into test and validation subsets with 50% of the subjects in each (N = 110). The test and validation subsets were thus composed respectively of 917 and 927 maps.
AutoEncoder performance Reconstruction performance of the CAEs is presented in Table  2. When comparing the 2 CAE architectures (4 layers vs. 5 layers) trained on the NeuroVault dataset, the mean correlations between original and reconstructed maps were better for the 4-layer architecture (86. 9% vs. 77. 8%). These results suggest that the reconstruction capabilities of the CAEs are dependent on the model architecture and the size of the latent space. Fig. 4  shows the reconstruction of a statistic map randomly drawn from the NeuroVault test dataset with the 2 CAE architectures. With the 4-layer architecture, details of the map were better reconstructed than with the 5-layer architecture (see the green square on the map). This was due to the level of compression of the data that was higher in the 5-layer CAE and that learned only the most useful features with less emphasis in learning specific details.
Choice of hyperparameters for HCP dataset Performance of the different models trained with the different hyperparameters can be found in Supplementary Table  S1 , available at  [52]. For the default algorithm initialization, the best model had 4 layers and was trained with a batch size of 32 for 500 epochs. This model achieved an accuracy of 90. 8% on average of the 5 folds of cross-validation. For the pretrained CAE initialization, the best model had 5 layers and was trained with a batch size of 64 for 200 epochs (average accuracy of 91. 8%). The best hyperparameters for each type of initialization (default and pretrained) were used in all subsequent experiments.
Choice of hyperparameters for BrainPedia dataset Results for all sets of hyperparameters are available in Supplementary Table  S2 , available at  [52]. For the default algorithm initialization, the model that achieved the best performance had 5 layers and a batch size of 64 for 500 epochs. This model classified the BrainPedia dataset with an average accuracy of 67. 1% and an average F1-score of 61%. The performance of the pretrained CAE was the best using a 5-layer architecture, a batch size of 64, and a training time of 200 epochs.
Benefits of self-taught learning on a heterogeneous dataset Table  5  summarizes the results for the classification of mental concepts on the small and the large BrainPedia datasets. These results are illustrated in Fig. 8. On a the small BrainPedia dataset, pretraining improved the performance of the classifier. When looking at the mean accuracies, respectively 56. 8% and 64. 5% for the classifier initialized with the default algorithm and the pretrained classifier, the difference was high (almost 8% of improvement). But in this case, the F1score was a better metric to assess the performance. Indeed, this metric focuses more on classification errors and is a better indicator of performance when classes are imbalanced, which was the case in this dataset in which some classes were more represented than others (e.
g. , in the small BrainPedia training set, 205 maps corresponded to the class "visual words, language, visual," whereas only 19 are in the class "left foot, visual"). When focusing on this metric, the pretrained classifier performance was markedly higher than the ones of the classifier with default initialization (11. 5% of improvement in mean F1-score). Performance (accuracies and F1-scores) was both significantly improved with the pretrained model compared to the default one (P < 0. 05). On the global BrainPedia dataset, performance also increased with pretraining. Mean accuracy and F1-score were higher for the the pretrained model (F1-score of 73. 6% against 64. 9% for the model with default initialization) even if the sample size of the dataset was higher and more classes were represented.
Features To better understand the behavior of each model-in particular, on what features they based their predictions on-we visualized the mean features across subjects of each layer of the pretrained, default models and baseline CAE for each class label (i. e. , contrast). Specifically, we studied the mean feature maps obtained across subjects in the test set (fold 1) of the N = 50 sample of the HCP dataset for different contrasts. This configuration was chosen due to the large difference between performance of default and pretrained models on this classification task. Our main interest was to see if the model would focus on general patterns of activation or more individual features. We focused on the contrasts that led to the most difficult classification tasks (i. e. , had the lowest per-class accuracy [less than 80%]).
Summary of the results In this work, we showed the benefits of self-taught learning with a large and variable database on the classification of 2 large public datasets with different sample sizes and classification tasks. In all cases, pretraining a classifier with an unsupervised task (in our case: reconstruction) was beneficial, but the level of improvement varied depending on the classification task and the size of the training dataset. When sample sizes were small, pretraining always improved the classification performance, regardless of whether the dataset was homogeneous or heterogeneous and the complexity of the classification task. In medical imaging, where the dimensions of the data are often very large and few samples are typically available due to high financial and human costs, learning a good representation of the data can be very difficult  [12, 59].
Another benefit of self-taught learning we found was the reduction of the training time. Performance of the pretrained classifier was better even with less training epochs. This was the case for both dataset results, which were computed for 500 epochs for the default algorithm and 200 epochs for the pretrained model. This is in line with  [60]  in which researchers showed that the pretrained models remain in the same basin of the loss function when trained on new data, and since the weights are already initialized close to a good representation of data, less epochs are necessary to adapt this representation for classification. Architectures of the models also had an impact on the benefits of self-taught learning. With both datasets, pretrained models performed better using the 5-layer architecture.
Deeper pretrained layers had more impact on classification performance, meaning that the features extracted by these layers were different from those learned by layers initialized with the default algorithm. In particular, the third and fourth convolutional layers showed the best benefits when being transferred, due to the generalizability of the extracted features. This was not the case for the fifth layer, for which features need to be specific to the classification task. The pretrained model improved the performance in terms of classification due its ability to focus on more generalizable features. By pretraining a model on a large variable dataset such as NeuroVault, we built a model that is less sensitive to the training data and less sensitive to individual differences, thus more generalizable and applicable to new subjects.
Limitations and perspectives The main limitation of our work is the classification experiments and datasets we chose. In fMRI, the number of possible labels and, thus, classification tasks is very high due to a lack of consensus in the field with respect to standardizing tasks, contrasts, and mental concepts  [38]. In our experiments, we used the labels provided by NeuroVault as specified in the original studies  [46, 48]. We chose to compare multiple types of classification on the HCP dataset to illustrate different approaches are in use in the field or that were used by other studies  [28, 29]. For BrainPedia, a multilabel decoding was performed in the original study since multiple concepts are associated with most maps. Labels we had access to were then the list of labels associated with each map.
To be able to compare our results with those of the homogeneous dataset (HCP), we chose to classify these as unique labels, which was less complex and less precise in practice. This type of issue is due to the lack of harmonization in the way tasks and cognitive processes are defined. Using ontologies such as Cognitive Atlas  [38] , NeuroVault annotations could be harmonized and enriched, as it was done by  [37]  by mapping the original labels to target ones from Cognitive Atlas or  [61]  in which cognitive conditions were annotated by a group of expert using the same atlas. In neuroimaging, many sources of variability can impact the results of an experiment and the generalizability of the results. Here, we investigated the generalizability of our model by assessing the benefits of pretraining on a heterogeneous dataset (BrainPedia).
While this dataset was heterogeneous in terms of the studies that were included, all maps were obtained using the same processing pipeline. Multiple studies have shown that the exact pipeline used to obtain an fMRI result can have a nonnegligible impact on fMRI statistic maps  [62, 63]. In the future, investigating performance of classification on a more variable target dataset with statistic maps from different studies but also processed using different pipelines would be of great interest. In a recent study  [10] , the authors tried to compare the performance of different classifiers trained on fMRI 3D volumes series obtained with various scenarios of minimal preprocessing pipelines. A similar experiment was recently made by  [64] , who found that preprocessing pipeline selection can impact the performance of a supervised classifier.
Comparing the adaptation capacities of models on volumes preprocessed with different pipelines could be also interesting to evaluate the impact of analytical variability on deep learning with fMRI and to see if the generalizability of our pretrained models also works for interpipeline differences. Note that self-supervised (instead of self-taught) learning could have also been used to pretrain our model, as it was done by  [33] , who designed self-supervised learning frameworks, inspired by the field of natural language processing, to pretrain mental state decoding models. Self-supervised learning is a supervised machine learning setting where the supervision is generated directly from the data and the model is pretrained using a supervised surrogate task. Self-supervised learning is particularly relevant if the surrogate task is close to the final one targeted by the user (e.
Due to the high computational time required to train a model, we only compared 2 model architectures (4 and 5 layers). Indeed, training a CAE model can be very time-consuming, particularly in our case since we use a large training dataset (N = 22,772) and high-dimensional data (k = 48 * 56 * 48). With the 4-layer model, for 200 epochs, it took approximately 48 hours to train on 1 GPU.
Acknowledgments This work was partially funded by  Region Bretagne  ( ARED MAPIS ) and  Agence Nationale pour la Recherche  for the program of doctoral contracts in artificial  intelligence  (project  ANR-20-THIA-0018 ). We thank  Gregory Kiar  who worked on a preliminary version of the autoencoder based on NeuroVault data. Data used in the HCP dataset were provided (in part) by the  Human Connectome Project, WU-Minn Consortium  (Principal Investigators:  David Van Essen  and  Kamil Ugurbil ;  1U54MH091657 ), funded by the 16  NIH Institutes  and Centers that support the  NIH Blueprint for Neuroscience Research , and by the  McDonnell Center for Systems Neuroscience at Washington University.
Data Availability Supporting data, including supplementary materials (figures and tables) and additional links (code and derived data) is also available via the GigaScience repository, GigaDB  [52]. The data used in this study are openly available on NeuroVault  [36]. No experimental activity involving the human participants was made by the authors. Only publicly released data were used. NeuroVault IDs of statistic maps included in each dataset (Neu-roVault, HCP, and BrainPedia) are available in the Derived Data section.
7 ABFigure 3 : 73 Figure 3:Overview of the process used to split the datasets for cross-validation. (A) The method performed for the HCP dataset and its subsamples. (B) Method used for BrainPedia and Small BrainPedia datasets. In both cases, the global dataset is first split into 2 subdatasets: "validation" and "test" with respectively 50% of the subjects, and then each subdataset is divided into 5 folds for cross-validation.
Introduction The poster focuses on drug prescribing, which is an old known problem in Public Health Informatics, using Pubmed as a supplementary resource providing more up to date therapeutic advances. We tackle the integration of heterogeneous Semantic Knowledge Resources SKR at several levels by their format but also by their representation within vocabularies available in the Linked Open Data  [1]  LOD. Its life-science part is full of SKRs about symptoms, indications, drugs and so on, persisted and defined by standardized vocabularies such as RDF  [2] , OWL  [3]  or SKOS  [4]. These schemes are natural tools to bring meaning to concrete data that would not be linked at first sight.
Methods We have to seek external knowledge in order to refine a MeSH (Medical Subject Heading  [5] ) query to be performed on Pubmed. The Romedi resource  [6]  provides us enough metadata by structuring the open Public Drug Database and marketed in France  [7]  in an RDF format. Romedi provides links between a marketed drug and its therapeutic indications from the ATC (Anatomical Therapeutic Chemical Classification  [8] ) and also their bibliographic tag to MeSH terms themselves represented as SKOS-based terminologies. K-Ware is the framework used for handling each of the RDF-based SKR, provides ways to: 1. Recognise a drug name from a Literal as an identified entity within Romedi 2. Browse a marketed drugs' attributes from Romedi to retrieve its active substance as an ATC entity 3. Handle mappings between MeSH terms and ATC codes Then establishing a cooccurrence query with all the MeSH terms retrieved from a starting prescription's expanded entities in order to retrieve PubMed's abstracts.
Conclusions K-Ware being more considered as an RDF framework rather than a simple terminoontological resource repository, allows by using its internal metamodel, the heterogeneous management of the Romedi vocabulary which is an RDF database along with the ATC and MeSH vocabularies which are rather SKOS terminologies without having to modify them. The K-Ware framework, by allowing us to move towards less data, but more metadata provides ways to not distort any represented Semantic Knowledge Resources for a welldefined project context.
Back-end automatic code corrector: Autocorrect software The corrector is a script that is launched in its sandbox for each code submitted. It starts by compiling the code and checks that there are no error or warning. If an error occurs, the analysis is stopped. Otherwise, the analysis continues, but if there is a warning, it is signaled to encourage students to correct it even if the result is correct. After the compilation step, the corrector runs the obtained program with the different datasets and tests associated with the corresponding exercise. Some are randomly chosen from a set of tests to prevent brute-forcing. For each test, the program output is compared with the expected output. These must be strictly identical to validate the exercise. A single failure causes the whole exercise to fail.
Using Baldr The implementation in Java of the previous theoretical work was carried out by our colleague Hubert Wassner. This cross-platform open source software is called Baldr, in reference to the Scandinavian god of light and truth. Its technical operating principle is explained by  Wassner (2014). In this section, we mainly focus on the educational benefits of this software. Analyzing a rendering of several tens or hundreds of files is quite simple: specifying the directory containing them for automatic processing is the only requirement. The analysis is very fast (on the order of a second). The first result displayed is a histogram of the inter-file similarity distances (see  Fig. 10, top). This graph allows us to quickly get an overall idea of the risk of fraud and its extent. Indeed, as the most suspicious cases correspond to the smallest distances, we can focus on the leftmost columns of the histogram.
For this, an early live demonstration of the effectiveness of the Baldr tool in front of all the students is done. 93% of students find that this demonstration convinced them that it is easier to work without cheating than to cheat without being detected. Thanks to this effort of prevention, transparency and systematic use of the tool, attempts at fraud are almost non-existent. So that the time spent working is faced in pleasant conditions for the students, we have taken particular care to make the educational platform fun and addictive. 67% of the students appreciate this point. It should explain why the platform was adopted very quickly and rather perceived as a serious game. For 83% of students the discovery of new visuals as they progressed had a dynamic effect. 74% of students say the short delay between code submission and interface feedback helps them maintain their attention.
, 2003; Han et al. , 2006; Ge et al. , 2013). Scientists have harnessed several strategies for developing novel drugs against  COVID-19 (Zumla et al. , 2016). The first strategy was to screen existing broad-spectrum anti-viral drugs such as ribavirin and cyclophilin. This approach is advantageous because the pharmacokinetic profile of the anti-viral drugs and their associated side effects have been clearly stated. One of the disadvantages of broad-spectrum anti-viral drugs is their nonspecificity, which might, in turn, result in low potency against coronavirus  (Chan et al. , 2013; de Wilde et al. , 2014). The second techniques involve screening molecular databases such as the ZINC database to identify potential anti-coronavirus compounds via high-throughput virtual screening. This approach has been used to develop biologically active compounds, lopinavir/ ritonavir, as anti-HIV agents  (de Wilde et al.