dataesr/scientific-paragraphs-categorization huggingface dataset:
The baseline was set from -2 seconds to -0. 5 seconds before spindle onsets. Given the a priori hypothesis that power increases are expected in the spindle frequency band in comparison to the baseline, we performed a one-tailed t-test with a significant threshold set at 0. 05. Statistical maps were then corrected for multiple comparisons using the Benjamini-Hochberg procedure to control the false discovery rate (Ntest = 60020)  (Benjamini and Hochberg, 1995). All analyses were done using the MATLAB R2019b software from The MathWorks  (Natick, MA)  and the open-source Brainstorm software  (Tadel et al. , 2011).
Code availability Sleep EEG data were processed using the MATLAB R2019b software from The MathWorks (Natick, MA) and the open-source Brainstorm and EEGLAB software. The codes for the detection and clustering of sleep spindles are available at the following GitHub repositories: https://github. com/arnaudboutin/Spindle-detection and https://github. com/arnaudboutin/Spindle-clustering. The codes used to perform other analyses are available from the corresponding author upon reasonable request.
Logical reasoning on natural language is one of the most challenging tasks for deep learning models. There has been an increasing interest in developing new benchmarks to evaluate the reasoning capabilities of language models such as BERT. In parallel, new models based on transformers have emerged to achieve ever better performance on these datasets. However, there is currently no library for logical reasoning that includes such benchmarks and models. This paper introduces LogiTorch, a PyTorchbased library that includes different logical reasoning benchmarks, different models, as well as utility functions such as co-reference resolution. This makes it easy to directly use the preprocessed datasets, to run the models, or to finetune them with different hyperparameters. LogiTorch is open source and can be found on GitHub 1.
, 2020 Hossain et al. , , 2022;; Helwe et al. , 2021). Hence, the question of whether these models can logically reason on 1 https://github. com/LogiTorch/logitorch text is still open  (Niven and Kao, 2019; Helwe et al. , 2021). New models are being created incessantly (e. g. , LogiGAN  (Pi et al. , 2022)  and Logiformer  (Xu et al. , 2022)  in 2022), and new datasets are being created to evaluate these models, including, e. g. , LogiQA  (Liu et al. , 2021b)  and ProofWriter  (Tafjord et al. , 2021). The initiative of open-sourcing toolkits has accelerated the progress in the field of natural language processing, driven by projects such as Transformers  (Wolf et al. , 2020)  from HuggingFace and Stanza  (Qi et al. , 2020)  from Stanford. However, this progress has not yet arrived in the field of logical reasoning: researchers still have to find and download different models, parameterize them, find the corresponding datasets, bring them into suitable formats, and fine-tune the models.
The datasets are maintained on different Web pages, exhibit different formats (JSON vs. full text, numerical vs. textual labels, etc. ), and follow different conventions, which makes it cumbersome to apply one model across several sources. The models themselves are implemented in different frameworks, have different input and output formats, require different dependencies, and differ in the way of running them, which makes it burdensome to exchange one model for another. Some models are not even available online, but have to be re-implemented from scratch based on the diagrams in the scientific publications. All of this hinders reproducibility, re-usability, comparability, and ultimately scientific progress in the area. In this paper, we propose to bring the benefits of open source libraries to the domain of logical reasoning: we build a Python library, LogiTorch, that includes 14 datasets and 4 implemented models for 3 different logical reasoning tasks.
All models can be called in a unified way, all datasets of one task are available in the same standardized format, and all models can be run with all datasets of the same task. All models have been re-implemented from the research papers that proposed them, and they have been validated by subjecting them to the same experiments as the original papers, with comparable results. More models and benchmarks are in preparation. LogiTorch works on top of PyTorch  (Paszke et al. , 2019) , and uses the Transformers library. It also includes utility functions used for preprocessing, such as coreference resolution and discourse delimitation. The rest of the paper is organized as follows. Section 2 discusses the design and components of LogiTorch, and describes the datasets, utility functions, and models.
LogiTorch LogiTorch is our Python library for logical reasoning on natural language text. Figure  1  shows the tree structure of our library. It is built on top of PyTorch and consists of 5 parts: Datasets. We gathered different logical reasoning datasets that allow users to evaluate the reasoning capabilities of deep learning models on natural language. Once a dataset is called from LogiTorch, it is downloaded, and wrapped into an object that inherits the Dataset class of PyTorch. This means that all datasets are accessible via the same interface. We describe the datasets in detail in Section 2. 1. Data Collators. Different models require different preprocessing steps for the same data and same task: one model may work on numerical vectors, the other on textual input. Hence, we designed, for each pair of a dataset and a model, a data collator that brings the dataset into the format required by the model.
Utilities. Some models require supplementary features in addition to the input text. For example, the DAGN model  (Huang et al. , 2021)  requires the discourse structure of the input in order to create a logical graph representation of it. For such cases, LogiTorch provides different utility functions, most notably for discourse structure analysis, coreference resolution, and logical expression extraction, which we discuss in Section 2. 2. Models. LogiTorch provides several deep learning models that have been designed to perform logical reasoning tasks such as proof generation and textual entailment. For each model, we either provide an implementation from scratch, or a wrapper over its original implementation. For the LogiTorch datasets qa mcqa proof_qa te data_collators utilities models pl_models transformer-based models, we use the Transformers library from HuggingFace for the implementation of the models.
Datasets The current implemented datasets focus on evaluating the reasoning capabilities of deep learning models. They cover four tasks: Multiple Choice Question Answering (MCQA), Question Answering (QA), Proof Generation, and Textual Entailment (TE). Table  1  shows the task and the number of instances of each dataset. Let us now describe each task and the associated datasets. Multiple Choice Question Answering (MCQA) is the task of choosing the correct answer to a question from a list of possible answers. Here is an example taken from the LogiQA dataset  (Liu et al. , 2021b).
The sentence can then be rewritten as α → β. From this, we can infer by transposition that ¬β → ¬α, which corresponds to "If you are able to use a computer, you have keyboarding skills". The LReasoner model  (Wang et al. , 2022)  uses this utility function to extend the input with logical expressions. In LogiTorch, we developed a wrapper over the code provided by LReasoner for this purpose. Discourse Delimitation is the task of splitting a text into elementary discourse units (EDU). It is used for the rhetorical structure theory (RST), in which it is a tree representation of a text where the leaves are EDUs, and the edges are rhetorical relations. For example, "A signal in a pure analog system can be infinitely detailed, while digital systems cannot produce signals that are more precise than their digital unit" is split into two EDUs: "A signal in a pure analog system can be infinitely detailed", and "digital systems cannot produce signals that are more precise than their digital unit".
Models LogiTorch currently implements four models: RuleTaker (QA task)  (Clark et al. , 2021)  is a RoBERTa-Large model  (Liu et al. , 2019)  that has been finetuned first on the RACE dataset  (Lai et al. , 2017) , and then finetuned again for rule-based reasoning. The model takes as input facts and rules and a boolean question. The output is either True or False. The RoBERTa model has a similar architecture to BERT, but performs better on many NLP tasks. This is because it is pretrained for a longer period, with large batches, and on a larger dataset. The pretraining task is only the Masked Language Modeling (MLM) task, but the masked tokens are changed after each training epoch. ProofWriter (QA and proof generation)  (Tafjord et al. , 2021 ) is a T5 model  (Raffel et al. , 2020)  finetuned to perform rule-based reasoning.
It takes as input facts and rules and a question. The output is either True, False, or Unknown (if the trained dataset considers the open-world assumption). T5 is a text-to-text transfer transformer that was pretrained on a variety of NLP problems such as textual entailment, coreference resolution, linguistic acceptability, and semantic equivalence. PRover (QA and proof generation)  (Saha et al. , 2020)  is built on RoBERTa with three modules: the QA module, Node module, and Edge module. The QA module is responsible for answering a question as either True or False. The Node and Edge modules are responsible for generating proofs. The Node module predicts the relevant rules and facts used to generate the answer, and the Edge module predicts the link between two relevant facts and between a relevant fact and a relevant rule.
BERTNOT (TE task)  (Hosseini et al. , 2021 ) is a BERT model that is pretrained using the unlikelihood loss and knowledge distillation functions for the MLM task to model negation. Then it is finetuned on textual entailment tasks. This model is more robust on examples containing negations, and performs better on the negated NLI dataset than the original BERT. Future releases will include newer models such as LReasoner  (Huang et al. , 2021) , Focal Reasoner  (Ouyang et al. , 2021) , AdaLoGN  (Li et al. , 2022) , Logiformer  (Xu et al. , 2022) , and LogiGAN  (Pi et al. , 2022).
Library Usage Listing 1 shows a detailed example of how a model can be trained on a rule-based reasoning dataset for QA. The RuleTaker model is trained on its corresponding dataset. In Lines 9-10, we initialize the training and validation datasets with the Rule-TakerDataset. We specify which sub-dataset and which split we want to use. In Line 12, we initialize the RuleTaker data collator for preprocessing the datasets. We then use the Dataloader to pre-load the datasets and use them as batches. In Line    terparts (Hossain et al. , 2020). All models use the same settings as in the original papers. Table  2  shows the results of the three different models on the QA task at different reasoning depths. Our model implementations achieve nearperfect accuracies, which are comparable to the performance in the original papers.
Conclusion We have introduced LogiTorch, a Python library for logical reasoning on natural language. It is built on top of PyTorch in combination with the Transformers and PyTorch Lightning libraries. LogiTorch includes an extensive list of textual logical reasoning datasets and utility functions, and different implemented models. The library allows researchers and developers to easily use logical reasoning datasets and train logical reasoning models with just a few lines of code. The library is available on GitHub and is under active development. For future work, we will add new datasets, and implement models such as DAGN, Focal Reasoner, and LogiGAN with their utility functions for feature engineering. Finally, we want to invite researchers and developers to contribute to Logi-Torch. We believe that such a library will lower the hurdles to research in the area, foster re-usability, encourage comparative evaluation, strengthen reproducibility, and advance the culture of open software and data.
: Context: David knows Mr. Zhang's friend Jack, and Jack knows David's friend Ms. Lin. Everyone of them who knows Jack has a master's degree, and everyone of them who knows Ms. Lin is from Shanghai. Dataset Task Training Instances Validation Instances Testing Instances AR-LSAT MCQA 1,630 231 230 ReClor MCQA 4,368 500 1,000 LogiQA MCQA 7,376 651 651 RuleTaker QA 587,922 84,030 173,496 ProofWriter QA/Proof Generation 585,860 85,520 174,180 ParaRules Plus QA 360,000 64,658 10,798 AbductionRules QA 80,024 11,432 22,928 ConTRoL TE 6,719 799 805 SNLI TE 550,152 10,000 10,000 MNLI TE 392,702 20,000 20,000 RTE TE 2,490 277 3,000 Negated SNLI TE - - 1,500 Negated MNLI TE - - 1,500 Negated RTE TE - - 1,500 Question: Who is from Shanghai and has a master's degree? Choices: (A) David (B) Jack (C) Mr. Zhang (D) Ms.
, 2021), which was designed similarly to the RuleTaker datasets. However, the ProofWriter which all require reasoning capabilities to choose the correct answer: AR-LSAT (Zhong et al. , 2021 ) is a dataset that was constructed by selecting the analytical reasoning section of 90 LSAT exams from 1991 to 2016. LogiQA (Liu et al. , 2021b)   assesses the logical deductive ability of language models for the case where the correct answer to a question is not explicitly included in the question. The corpus includes paragraph-question pairs translated from the National Civil Servants Examination of China. ReCloR (Yu et al. , 2019)   is a corpus consisting of questions retrieved from standardized exams such as LSAT and GMAT. To adequately evaluate a model without allowing it to take advantage of artifacts in the corpus, the testing set is split into two sets: the EASY set where the instances are biased, and the HARD set where they are not.
rules and a boolean question. The model has to perform logical deductions from the rules and facts in order to answer the question. The dataset includes synthetically generated subsets that require different depths of reasoning, i. e. , different numbers of deduction steps to answer a question. The dataset also includes the Bird dataset (which showcases McCarthy's problem of abnormality (McCarthy, 1986) ), the Electricity dataset (which simulates the functions of an appliance), and the ParaRules corpus (where crowd workers paraphrased sentences such as "Bob is cold" to "In the snow sits Bob, crying from being cold"). ParaRules Plus (Bao, 2021 ) is an improved version of ParaRules (Clark et al. , 2021). It has more examples for the instances with larger reasoning depths. AbductionRules (Young et al.
, 2022)   is a dataset that evaluates the abductive reasoning capabilities of language models. It is generated similarly to ParaRule Plus, but in this task, the model has to generate an answer to explain an observation. dataset contains proofs for the answer of each question. Furthermore, there is a variant of the dataset that considers the open-world assumption. Textual Entailment (TE, also RTE) is the task of predicting whether a premise entails or contradicts 1 import pytorch_lightning as pl 2 from pytorch_lightning. callbacks import ModelCheckpoint 3 from torch. utils. data. dataloader import DataLoader 6 from logitorch. datasets. qa. ruletaker_dataset import RuleTakerDataset 7 from logitorch. pl_models. ruletaker import PLRuleTaker context = "Bob is smart. If someone is smart then he is kind.
05) or low frequency (0. 01 ≤ MAF ≤ 0. 05) variants. Analyses were repeated in 76,202 individuals from UKB conditioning on independent significant variants identified in the corresponding European GWAS meta-analysis, and residing within the same locus as the gene. These conditional analyses showed that associations for KCNQ1, KCNH2, and RNF207 with QT and/or JT were independent of flanking variants identified by GWAS (Supplementary Data 11). Because conditional analyses required a large sample with a shared set of variants, we lacked adequate power to definitively determine the independence of DLEC1 from nearby common variants. For MYH7 and TNNI3K, there were no GWS variants at the locus in our single-variant meta-analysis.
Tissue-and cell-type specific effects of variants through regulatory elements Potential target genes of regulatory variants were identified using two long-range chromatin interaction datasets (40 kb-resolution Hi-C and ~4 kb-resolution promoter-capture Hi-C) from LV and RV tissue  34, 35. Promoter interactions were identified at 39 (22. 2%) QT loci (Supplementary Data 14a and 14b). Evaluation of cardiac cell-type specific effects using single nucleus Assay for Transposase-Accessible Chromatin using sequencing (snATAC-seq) data  36  , identified significant enrichment at open chromatin regions for QT in atrial and ventricular cardiomyocytes (Supplementary Fig. 5 ). Promoter interactions were identified at 46 (29. 7%) JT and 28 (23. 1%) QRS loci (Supplementary Data 14a and 14b). Cardiac cell-type specific enrichment was significant in atrial and ventricular cardiomyocytes for both JT and QRS, and in adipocytes for JT (Supplementary Fig.
5 ). In the Reactome database, top pathways were related to signal transduction, including protein kinase B mediated events, mechanistic target of rapamycin (mTOR) signaling, the phosphoinositide 3-kinases cascade and the insulin receptor signaling cascade (Supplementary Data 16). The top 10 mouse phenotypes enriched for these gene sets included abnormal myocardial layer/trabeculae morphology, decreased embryo size/growth retardation, and increased heart weight. GO biological processes significant for JT (and QT), but not QRS, included regulation of gene expression, histone and chromatin modification, insulin receptor signaling and response to insulin stimulus (Fig. 5 , Supplementary Data 16). Cellular growth, the transmembrane receptor protein serine/threonine kinase signaling pathway and vasculogenesis were enriched only for QRS.
Identification of potential drug targets for therapeutic opportunities To identify potential drug targets for arrhythmia, we interrogated the druggable gene-set database published by Finan et al. 40. We examined all 200, 173 and 155 plausible candidate genes from the QT, JT and QRS multi-ancestry meta-analyses respectively, including known and previously unreported loci. 53 (QT), 46 (JT), and 31 (QRS) genes were identified as potential drug targets, that are not current targets of antiarrhythmic drugs (Supplementary Data 20). Of these, 21 QT, 17 JT, and 10 QRS genes were classed as Tier 1, encoding proteins that are targets of drugs either approved or in development. Genes from significant signals in cardiac tissue-specific eQTL co-localization and Hi-C analyses may be favored for prioritization.
While cohorts have extracted ECG parameters using different methods, we believe the large sample sizes and averaging of effect estimates during meta-analysis should limit the impact of any variability on our findings. Of note, we did not observe substantial heterogeneity across results from previously unreported variants. Future GWAS meta-analyses could use the same algorithm across all cohorts to extract ECG phenotypes, but raw digitalized data are not available for all participants of the current study, so we were unable to do this without substantially reducing the total sample size. Our study includes extensive in-silico follow-up of variants; however, it does not identify causal relationships. Functional follow-up is warranted using the latest advances, including single-cell genomics to further evaluate the relationship of variants with gene-expression  82  and gene-editing tools (e.
Study cohorts A total of 35 studies (involving 53 ancestry-specific sub-studies), including members of the Cohorts for Heart and Aging Research in Genomic Epidemiology (CHARGE) consortium  85  contributed to this study (Supplementary Data 1). A maximum total of 252,977 individuals of European (84%), Hispanic (7. 7%), African (6. 7%), South and South-East Asian (<1%) ancestries were included. All participating institutions approved this project and informed consent was obtained for all individuals at the study level. Cohorts were predominantly populationor community-based with a small number of studies ascertained on a specific case status. Affymetrix or Illumina arrays were typically used for genotyping. Study-specific genotype quality control filters prior to imputation, including call rate, Hardy-Weinberg equilibrium (HWE) Pvalue, and minor allele frequency, are provided in Supplementary Data 2.
All GWAS summary data utilized NCBI build 37. Most studies imputed ungenotyped rare and common variants using 1000G reference panels (40/53 sub-studies); the remainder used the Haplotype Reference Consortium (HRC) panel (r1. 1 2016), (Supplementary Data 2)  86, 87. Fourteen studies (including 31 sub-studies), imputed genotype data for X chromosome analysis. The pooled multi-ancestry sample size for X chromosome analyses was 86,600 (75,607 European, 7040 Hispanic, 1943 African, 709 South Asian, 590 South East Asian and 711 mixed) for females and 60,343 individuals (52,070 European, 5182 Hispanic, 1518 African, 806 South Asian, 479 South East Asian and 379 mixed) for males.
Cohort-level single variant association analyses Single variant genome-wide association studies were performed by each participating cohort for QT, JT, and QRS. For each trait of interest, additive genetic models were implemented for two phenotypes: (1) the raw phenotype (on the millisecond (ms) scale)) and (  2 ) the rank-based inverse normal transformation of each phenotype (on the standard deviation scale due to non-normal distributions of these traits). Per study summary statistics for each ECG measure and covariate are provided in Supplementary Data 3. Individuals were excluded at the study level for: prevalent myocardial infarction or heart failure, pregnancy at the time of recruitment, implantation of a pacemaker or implantable cardiac defibrillator, QRS duration >120 ms, or right or left bundle branch block or atrial fibrillation on ECG.
The QRS duration criterion was used as a surrogate marker for bundle branch block and interventricular conduction delay that was not identified during ECG analysis. Additionally, if the data were available, individuals using digitalis, class I or III anti-arrhythmics or QT prolonging medication were excluded. These exclusions were chosen to reduce the risk of confounding in our analyses of ECG parameters, where the bulk of the power comes from normal variation of QT, JT, and QRS. This will have reduced the total sample size available, however the genetic contribution to ECG interval variation in these disease states could differ, warranting separate investigations. Cohorts including related individuals used appropriate software to account for this e. g BOLT linear mixed model software (BOLT-LMM)  88  (which fits a linear mixed model on hard-called genotyped single nucleotide polymorphisms (SNPs)) or other software incorporating a kinship matrix or pedigree  [89] [90] [91].
An imputation quality cut-off of Rsq >0. 3 (or similar in IMPUTE) was applied in all cohorts to ensure high quality variants were included in the meta-analysis. Covariates were included in the GWAS model and chosen for their known association with each ECG measure. These included age (years), sex (except in sex-stratified X chromosome analyses), RR interval (ms), height and body-mass index (BMI, kg/m 2 ). Genetic principal components (PCs) were included to account for cryptic population stratification except in cohorts with pedigree data available or when analyses were performed using linear mixed models. As there may be ancestral differences in ECG measure, cohorts comprised of multiple ancestries performed separate analyses for each ancestry to control for underlying population stratification.