dataesr/scientific-paragraphs-categorization huggingface dataset:
Step 2: Create folder structure, rename and copy NIfTI files BIDS relies heavily on a particular folder structure and naming scheme of files. We begin creating the folder structure, by creating one subfolder for each of the 29 subjects named "sub-01", "sub-02", "sub-03" etc. Inside each of the subject subfolders we create three subfolders: "anat" (for anatomical scans), "dwi" (for diffusion scans), and "func" (for task fMRI). Those names are not arbitrary and must follow the BIDS specification (see Supp. File 1). This dataset includes two anatomical scans per subject: high-resolution T1 weighted and in-plane T2 weighted. They need to be renamed to "sub-01_T1w. nii. gz" and "sub-01_inplaneT2. nii. gz" (respectively) and moved to the "anat" subfolder. This operation has to be repeated for all subjects.
Along the. nii. gz files. json files (with the same body of the file name) should be also moved. Similarly we move the diffusion files into the "dwi" folder. The naming scheme is analogous "sub-01_dwi. nii. gz". In addition to. json files we also move the. bvec and. bval files containing gradient information produced by dcm2niix. Finally we follow suit with the task fMRI files. This dataset includes four different tasks with the following names: stop-signal, Balloon analog risk task (BART), discounting, and emotion regulation, which we label as "stopsignal", "bart", "discounting" and "emotionregulation" correspondingly. The naming scheme for functional is "sub-01_task-stopsignal_bold. nii. gz" (where "01" is replaced by corresponding subject label for the other subjects and "stopsignal" is replaced by corresponding task label for the other tasks).
This data should be saved in a text file called "participants. tsv" in the root of the dataset directory. This file has one compulsory column: "participant_id" (for example "sub-01", "sub-02") and can include any number of other arbitrarily named columns describing participants. Optionally a "participants. json" file can be provided with description of each column and links to external ontologies (see Section 4. 2 in Supp. File 1). Step 4: Add missing metadata All of the metadata in. json files were so far obtained using the dcm2niix converter. In addition to this we need to provide the name of each fMRI task. Optionally we can add information about task instructions and description as well as link the tasks an external ontology such as Cognitive Atlas or Cognitive Paradigm Ontology. Metadata organization can also be simplified using the inheritance rule: Metadata fields common across all subjects can be specified in one JSON file in the root of the directory instead of being repeated for each subject (see Section 3.
5 in Supp. File 1 for details). Finally we need to create a dataset_description. json file with fields that include the name and description of the dataset as well as the version of BIDS standard used. This file can also be used to list authors and ways to reference the dataset (see Section 8. 1 in Supp. File 1). Step 5: Validate the dataset Once the dataset is assembled, the BIDS Validator can be used to check if any of the required or recommended metadata are missing. In addition the validator has built in heuristics to spot incorrect definitions of missing values (for example "NA" instead of "n/a"), use of wrong units (milliseconds instead of seconds), missing scans and inconsistent scanning parameters across subjects. The validator works in the Chrome web browser with no need to install additional software, and performs the validation on the client side (i.
e. , no data are uploaded or shared) so it is suitable for sensitive datasets that are not intended for public sharing. Any BIDS compatible dataset can be readily fed into MRIQC or QAP toolboxes (see Adoption) that calculate quality measures. Thanks to formal structure of BIDS no additional metadata are required as an input. Outputs of those quality analyses can be included along with the dataset (see Section 3. 4 in Supp. File 1). Similarly BIDS2ISATab tool can be used (again with no need to specify additional metadata) to generate metadata summary files that are required for publishing with Scientific Data and GigaScience journals.
Adoption Despite its relatively young age BIDS has been already adopted by the OpenfMRI repository  16. Since the switch to the new standard in December 2015, thirteen new BIDS compatible datasets have been published. In addition several software packages added support for BIDS: SciTran (database)  29  , Quality Assurance Protocol (QA toolboxhttps://github. com/preprocessed-connectomes-project/quality-assessment-protocol ), MRIQC (QA toolbox - https://github. com/poldracklab/mriqc ), and automatic analysis (workflow toolbox)  30  have added BIDS support. In addition, a number of tools have been developed to help working with BIDS datasets. Those include: bids-validator (a validation tool -https://github. com/INCF/bids-validator ), openfmri2bids (OpenfMRI convention to BIDS converter -https://github.
com/INCF/openfmri2bids ), BIDSto3col (FSL modelling helper tool - https://github. com/INCF/bidsutils/tree/master/BIDSto3col ), and BIDS2ISATab (tool for extracting ISA-Tab 31 compatible metadata from BIDS datasets to improve speed and accuracy of data curation journals using this standard -https://github. com/INCF/BIDS2ISATab ).
Discussion Since BIDS was designed to maximize adoption, it heavily relies on established file formats such as NIfTI and bvec/bval (see the protocol for details). This decision was made because those file formats are widely supported by neuroimaging software. Using other file formats (such as DICOM which is closer to the scanner output or HDF5 which is much more flexible and allows for storing all metadata) would result in a more concise and robust data structure, albeit at the cost of additional software development necessary to adapt existing software to the new file format. Storing metadata in JSON files has advantages of accessibility, but can be error prone because data and metadata do not live in the same file. In future revisions of BIDS we will explore the possibility of storing metadata as a JSON text extension of the NIfTI header.
While we chose NIfTI to store neuroimaging data due to its popularity, we also recognize that specific tools or communities use other neuroimaging file formats such as MINC or NRRD for both technical and historical reasons. Different flavors of BIDS can be designed to support such formats, which would additionally require (1) identification of the metadata fields that should be included in the sidecar JSON file (as opposed to the data file headers), and (2) modifying the validator to read the new file format and check for required and optional metadata. For example, the MINC community (represented in this work by authors SD and TG) is currently working on an mBIDS specification that is based on BIDS, but uses MINC file format instead of NIfTI. Even though having multiple flavors of one standard can be problematic and confusing for software developers, those flavors are also necessary to meet the specific needs of some communities.
To avoid confusion, any future derivatives of the BIDS specification that are not compatible with the original should be clearly marked in the dataset metadata. At the same time we acknowledge that NIfTI file format is far from perfect and it quite likely that it will be replaced by a solution more capable of random access to large compressed blocks of data and with built-in extensible metadata storage. Through initiatives such as mBIDS we can decouple file organization from a particular file format used for storing imaging data. The current release of BIDS does not include support for Electroencephalography (EEG) and Magnetoencephalography (MEG) data, because, at present, there is no single commonly accepted data exchange file format for such data (akin to NIfTI in neuroimaging). However, we plan to extend the standard with support for EEG/MEG in a future release.
Similarly the current version of the standard does not cover Positron Emission Tomography (PET), Arterial Spin Labeling (ASL) and spectroscopy, but those extensions will be considered in the future. Our major focus in the near future will be on extending the software ecosystem supporting BIDS to provide incentives for researchers to use it. Work is underway to include BIDS support within heudiconv (data organizer), PyMVPA (statistical learning toolbox), automatic analysis (framework for analysing multimodal datasets), C-PAC (resting state analysis toolbox), and CBRAIN (data analysis platform). Furthermore, XNAT, COINS and LORIS databases are planning to support BIDS as an export and/or import option. We also plan to build tools to facilitate conversion to the NIMH Data Archive. It is also important to acknowledge that formal, machine readable descriptions of datasets such as BIDS can only complement, but not replace free form descriptions written in prose such as data papers or data descriptors 32.
Such unconstrained format is not only capable to capture motivation behind acquiring a particular dataset or describe in detail experimental design of a particular behavioural paradigm, but also can be easily turned into an academic paper providing an important incentive for data sharing. Adoption of BIDS in data papers describing neuroimaging data can increase their value, because BIDS will make it easier to assess completeness and internal validity of a dataset and make it easier to reuse. This article serves only as an introduction to the BIDS standard -the complete version of the the specification is available as supplementary material. In addition, for example datasets, a list of resources, and pointers on how to give feedback on future releases please visit http://bids. neuroimaging.
MethodsWork on the Brain Imaging Data Structure began at a meeting of the INCF Neuroimaging Data Sharing Task Force ( wiki. incf. org/mediawiki/index. php/Neuroimaging_Task_Force ) held at Stanford University on January 27-30th 2015. While a flexible solution using the PROV W3C model ( http://www. w3. org/TR/prov-overview/ ) was first investigated, it was acknowledged that this technology would be only viable if tools were in place to write the associated metadata. Since experimental data are obtained from multiple tools, a solution accessible to most neuroimaging researchers was designed. An initial draft was heavily inspired by the data structure used by the OpenfMRI database, but soon evolved beyond backward compatibility. After the initial draft was formed, a series of discussions and public calls for feedback were conducted.
Feedback was solicited over Twitter, by presenting BIDS and distributing informational pamphlets at conferences (INCF, SfN), as well as by sending emails to SPM, FSL, Freesurfer, MRTrix, Slicer, Nipy, and HCP mailing lists (reaching over 5,000 researchers). Further refinement of the standard was facilitated by a meeting held during the OHBM conference in Honolulu in June 2015. The discussion over the standard involved domain researchers, computer scientists, MRI physicists, methods developers (FSL, SPM, Slicer, Nipy, PyMVPA, C-PAC, nilearn and aa), data curators (OpenfMRI, FCP/INDI, HCP, NKI, SchizConnect, ABIDE, DataLad, and BIRN) and database developers (COINS, LORIS, XNAT, NiDB and SciTran). The first Release Candidate was published on September 21st 2015 along with 22 example datasets, online and command line validation tools ( https://github.
ACKNOWLEDGEMENTS This work has been supported by the  International Neuroinformatics Coordinating Facility (INCF)  and by the  Laura and John Arnold Foundation. VDC has been supported by in part by  NIH   P20GM103472. TEN and CM have been supported by the  Wellcome Trust. NN has been supported by  NIH NIAAA  [ 1 U01 AA021697 ]. JBP, BNN, and RAP have been supported by  NIH   NIAAA OD [1 U01 AA021697-04S1 ]. DAH was supported by the  Intramural Research Program  of the  NIMH. MH was supported by the  German federal state of Sachsen-Anhalt  and the  European Regional Development Fund (ERDF) , project:  Center for Behavioral Brain Sciences (CBBS). T. A. was supported by the  Medical Research Council (United Kingdom)  [ MC-A060-53144 ]. YOH has been supported by in part by  NSF  [1429999].
Active learning is the problem of interactively constructing the training set used in classification in order to reduce its size. It would ideally successively add the instance-label pair that decreases the classification error most. However, the effect of the addition of a pair is not known in advance. It can still be estimated with the pairs already in the training set. The online minimization of the classification error involves a tradeoff between exploration and exploitation. This is a common problem in machine learning for which multiarmed bandit, using the approach of Optimism int the Face of Uncertainty, has proven very efficient these last years. This paper introduces three algorithms for the active learning problem in classification using Optimism in the Face of Uncertainty. Experiments lead on built-in problems and real world datasets demonstrate that they compare positively to stateof-the-art methods.
Evaluation In this section, we evaluate the algorithms introduced in the previous sections. Each of the evaluation will be the scope of a comparison between our algorithms and state-of-the art algorithms. Algorithms are evaluated on two different benchmarks depending on the classifier used. In the case of independent clusters, either the instance space is already discrete, or it is continuous and must be partitioned. In the latter, the partition has to be chosen before taking any samples and cannot be changed after that. The predicted label in each cluster depends only on samples drawn in it. Thus, the number of clusters is limited by the budget of samples and by a desired good learning rate. This makes the partition rough and the classifier noncompetitive on real world datasets. However, the small number of parameters needed to represent any real world problem allows us to build a representative benchmark.
The performance of the algorithms closely depends on the choice of the partition. Clusters may regroup instances with a great dispersion of labels, either because the instances are subject to a lot of noise, or because the mean label varies rapidly. In this second case, a better discretization of the instance space causes a better true risk when the correct label is given. But this implies increasing the number of clusters, and for the prediction of the label to be equivalently accurate, we need a higher budget. As the problem of the choice of the partition is not studied here, the use of our algorithm on real world datasets with a continuous instance space would not be competitive with methods designed for this problem. Instead, we show that, given any partition, the algorithm performs better than other algorithms with the same constraints.
In the experiments, we use a Gaussian kernel covariance matrix: 𝑃 𝑖,𝑗 = 𝑒 -|𝑥 𝑖 -𝑥 𝑗 | 2 /2𝑠 2. ( 63 ) The scale parameter  has been tuned to give the best performance with a full training set. We thus evaluate our algorithm on several real world datasets from the UCI Machine Learning Repository  [24]. The four datasets used in this paper are Australian, Diabetes, Heart, and Wdbc. In all those datasets, the instances belong to a continuous instance space. In each run of experiments, the dataset is randomly divided into two. The first half is used as the pool of unlabeled instances in which the algorithm is allowed to pick, while the second half is used as the test set. At each time step, the true risk of the current prediction is estimated via the test set and recorded. The global performance of an algorithm at each time step is computed as the average of the performance among the runs.
In our experiments, the number of runs is set to 1. 000. The parameter  is tuned for every dataset using a grid search. We compare our algorithm with existing state-of-the-art method and some baselines. Random Sampling. This is the simplest baseline. At each time step a random instance is drawn from the pool of unlabelled instances. Full Knowledge. This is the best algorithm we can do. At each time step, an instance is selected according to the full knowledge criterion. The values of the parameters are used; thus, it is unrealistic but it serves to show how well the exploration/exploitation tradeoff is achieved. Uncertainty Sampling (see  [7] ). This is the most common active learning algorithm. At each time step, the instance which is most uncertain about how to label is selected. We built our optimistic algorithm by first defining a full knowledge criterion which guides the allocation of samples in the case where the true values of the parameters are known from the beginning.
Indeed, the myopic minimization of the risk has no guaranty to lead to the optimal allocation. For example, the best performances of the classifier at time step 2 could be achieved by the inclusion of a pair of instances that does not contain the instance that leads to the best performance at time step 1. Although the empirical results show that using a myopic minimization in full knowledge performs quite well, we can see that on the Wdbc dataset Figure  4 (d), for a short period around 55 labeled instances, uncertainty sampling achieves slightly better performance than the full knowledge allocation. The optimistic algorithm can not thus do better on this period. Still, we can see that it outperforms it on the 20 first samples as well as it keeps high performance on the end while Uncertainty Sampling seems to lose accuracy.
This last phenomenon can be explained. Let us look at Figure  5  where we display the results for the Wdbc dataset with the range of time steps extended to show the behavior of the algorithms until the last sample of the pool is retrieved. We can see the performance of all the algorithm decreases while approaching the end. We may think that the performance can only increase with the number of samples taken into account by the classifier. We know that this is not necessarily true, particularly if the classifier overfits. Also, in active learning, one can select a subset of instances that achieves better performance than when taking all the instances from the pool. The question of a stopping criterion has already been studied in order to avoid spending useless resources. Here, we see that it is even more crucial because it could lead to better performances.
In OEMAL, the criterion used represents the maximum decrease of the true risk one can expect by taking a particular sample. Thus, if the value of the criterion is less than 0, this means that there is a high probability (1 -) that the true risk will not decrease but increase. In this case, it is preferable not to sample this instance. But if the maximum criterion is less than 0, it is preferable not to sample at all. We use this as a stopping criterion. We compare the final performances of our algorithm with and without the stopping criterion for different datasets in Table  1. Two of the four datasets see their true risk increase at the end, Diabetes and Wdbc. We can see that the use of a stopping criterion is efficient in those cases and does not greatly alter the performances in other cases.
Evaluation. The matrix  appearing in the classifier we used so far is derived from the use of a set of partitions. The value in each cell of the matrix corresponds to the weight of each estimation in each prediction. We saw that it was possible to use clusters of the thinnest partition that contain only one instance from the pool. Instances are linked to others through. Inherently, in this case  fully specifies the data manifold structure. The theory that leads to OEMAL is built upon this set of partition,   6. We can see that OEMAL-k always does better than uncertainty sampling. Even in the Wdbc dataset where uncertainty sampling performs worse than random sampling, OEMALk manages to get better performance. One important thing in active learning is the choice of the classifier. With a wellfitted classifier, even the random sampling strategy could perform better than the best active learning strategy on a poor classifier.
It is thus convenient that our algorithm is not limited only to one kind of classifier and can easily generalize to kernel classifiers. For example, on the Wdbc dataset Figures  4(d ) and 6(d), the random sampling strategy performs significantly better when using a kernel than a set of partitions, and OEMAL-k keeps improving the results. On the other hand, on the Diabetes dataset Figure  6 (b), the random sampling strategy is also better with the kernel classifier, but OEMAL performs better than OEMAL-k. As we have seen in Figure  5 , the active learning strategy of the best performance for the classifier may be achieved with only a subset of instances. Maybe the former classifier had a better potential than the last. This is confirmed by the performance of the full knowledge criterion. In this section, we evaluated the performance of OEMAL which is built on this approach on several real world datasets.
Conclusion In this paper, we show that the problem of active learning in classification can be studied through the eye of Optimism in the Face of Uncertainty. This has the advantage to allow the selection criterion to be defined as close as possible to the evaluation function. It introduces three error minimization algorithms which use this approach. The experiments, conducted on built-in problems as well as real datasets,  show that they perform comparatively well to state-of-theart methods. An extension of the last algorithm shows that it can be generalized to other kernels. We however constrained the matrix  to remain the same all along the progress of the algorithm. Our perspective is to work with a changing matrix  such as in kernel regression or Gaussian processes where the variance of the estimates is given.
Acknowledgements We thank  Fabrizio Adriani ,  George-Marios Angeletos ,  Steven Durlauf ,  Miltos Makris ,  Eric Maskin ,  Peyton Young  and seminar participants at the  Universities of Birmingham, Bolzano-Bozen, Bristol, Exeter, Halifax, Keele, Lausanne, Louvain-la-Neuve, Milan, Royal Holloway and Vienna  for useful comments and discussions. We are also indebted to the Editor,  Christian Hellwig , and two anonymous referees for valuable suggestions. All errors are our own.
ABSTRACT Life sciences are yielding huge data sets that underpin scientific discoveries fundamental to improvement in human health, agriculture and the environment. In support of these discoveries, a plethora of databases and tools are deployed, in technically complex and diverse implementations, across a spectrum of scientific disciplines. The corpus of documentation of these resources is fragmented across the Web, with much redundancy, and has lacked a common standard of information. The outcome is that scientists must often struggle to find, understand, compare and use the best resources for the task at hand. Here we present a community-driven curation effort, supported by ELIXIR--the European infrastructure for biological information--that aspires to a comprehensive and consistent registry of information about bioinformatics resources.
The sustainable upkeep of this Tools and Data Services Registry is assured by a curation effort driven by and tailored to local needs, and shared amongst a network of engaged partners. As of November 2015, the registry includes 1785 resources, with depositions from 126 individual registrations including 52 institutional providers and 74 individuals. With community support, the registry can become a standard for dissemination of information about bioinformatics resources: we welcome everyone to join us in this common endeavour. The registry is freely available at https://bio. tools.