dataesr/scientific-paragraphs-categorization huggingface dataset:
One of the notable recent works addressing outcome identification as an entity extraction task, rather than sentence classification, is  (Blake and Lucic, 2015). The authors addressed a particular type of syntactic constructions -comparative sentences -to extract three items: the compared entities, referred to as the agent and the object, and the ground for comparison, referred to as the endpoint (synonymous to outcome). The aim of this work was to extract corresponding noun phrases. The dataset was based on full-text medical articles and included only the sentences that contain all the three entities (agent, object and endpoint). The training set comprised 100 sentences that contain 656 noun phrases. The algorithm proceeds in two steps: first, comparative sentences are detected with the help of a set of adjectives and lexico-syntactic patterns.
, 2014)  word embeddings and character-level representations  (Ma and Hovy, 2016) ; and a fine-tuned bi-LSTM using BERT  (Devlin et al. , 2018)  vector word representations. BERT (Bidirectional Encoder Representations from Transformers) is a recently introduced approach to pre-training language representations, using a masked language model (MLM) which randomly masks some input tokens, allowing to pre-train a deep bidirectional Transformer using both left and right context. The pre-trained BERT models can be fine-tuned for supervised downstream tasks by adding one output layer. BERT was trained on a dataset of 3. 3B words combining English Wikipedia and BooksCorpus. Two domain-specific versions of BERT are available, pre-trained on a combination of the initial BERT corpus and additional domain-specific datasets: BioBERT  (Lee et al.
We finetuned and tested the BioBERT model trained on the whole corpus, and both cased and uncased base models for BERT and SciBERT (using SciVocab). We did not perform experiments with BERT-Large as we do not have enough resources. We used the code provided by BioBERT for the entity extraction task  1. The relation extraction assumes that the entities have already been extracted and are given as an input to the algorithm, with the sentence in which they occur. To predict the tag for outcome -significance level pair, we use machine learning. As the first approach, we compared several classifiers available in the Python scikit-learn library  (Pedregosa et al. , 2011) : Support Vector Machine (SVM)  (Cortes and Vapnik, 1995) ; Deci-sionTreeClassifier  (Rokach and Maimon, 2008) ; MLPClassifier(von der  Malsburg, 1986) ; Kneigh-borsClassifier  (Altman, 1992) ;  GaussianProcess-Classifier (Rasmussen and Williams, 2005) ; Ran-domForestClassifier  (Breiman, 2001) ; AdaBoost-Classifier (Freund and Schapire, 1997); Extra-TreesClassifier  (Geurts et al.
, 2006) ; Gradient-BoostingClassifier  (Friedman, 2002). Feature engineering was performed manually and was based on our observations on the corpus. Evaluation was performed using 10-fold crossvalidation. To account for different random states, the experiments were run 10 times, we report the average results of the 10 runs. We performed hyperparameters tuning via exhaustive grid search (with the help of the scikit-learn GridSearchCV function). As the second approach, we employed a deep learning approach to relation extraction, finetuning BERT-based models on this task. We tested the same models as for the outcome extraction. We used the code provided by BioBERT for relation extraction task 2. The algorithm takes as input sentences with the two target entities replaced by masks ("@outcome$" and "@significance$") and positive/negative relation labels assigned to the sentence.
Objectives This study aims to identify the sources of funding for investigator-initiated clinical trials (IICTs) in Portugal, and to recommend ways to improve the quality of information collected from clinical trial databases about funding. Design and methods A systematic search of trial registrations over the last 13 years-using the WHO International Clinical Trials Registry Platform (WHO-ICTRP) and four clinical trials registries (CTRs)-was carried out to identify IICTs in Portugal, used as a case study. Data from the databases were compared with data contained in publications to evaluate the consistency of information on funding sources. The term 'database' is used in this study to refer to both the WHO-ICTRP and the CTRs. When mentioned separately, the WHO-ICTRP is referred to as a 'platform', while the CTRs are referred to as 'registries'.
Outcome Suggestions to improve clinical trials databases to clearly identify the funding sources and data ownership in IICTs. results Two hundred and eighty-two IICTs were identified in Portugal. Twenty per cent of trials were supported by industry with unclear information on the ownership of the results. Inaccuracy was found in the information about sponsors and funders. The information about funding in all resulting publications (77 out of 133 completed studies) was also inconsistent between databases in 35 out of 77 (45%) of the studies. Notably, 23% of the trials funded by non-profit organisations (n=226) received funds from international and/or national funding agencies. Conclusions Identification of IICT funding and ownership of results is unclear in the databases used for this study, which may lead to misunderstandings about the independence of the obtained results.
IICTs might include trials for the paediatric use of medicines authorised for adults, or comparison of commercialised medicines. IICTs may also include pragmatic trials, and repurposing studies for novel indications for registered medicines, including in the field of rare diseases. Besides medicines, IICTs can address medical devices, nutrition, behaviour and surgery procedures to provide evidence on performance or the effect on health and well-being of participants. When compared strengths and limitations of this study ► Provides an analysis of the types of funding of investigator-initiated clinical trials (IICTs) for the first time, promoting discussion on the need for transparency. ► Compares information about funding sources contained in clinical trials databases with that provided in publications.
► Provides the basis for specific recommendations to increase the quality of the information provided in databases. ► As a case study, data on funding sources are limited to studies initiated in Portugal. As such, the study may not be representative of other countries. ► The information on IICT funding (in the databases used) is incomplete, which may provide a partial or inaccurate portrait of results. Open access with trials using medicines, other interventions are less demanding in terms of regulatory and ethical approvals. In industry clinical trials, the funding source is the sponsor, whereas in non-commercial trials the sponsor might not be the funder. According to the European Regulation for clinical research (Regulation 536/2014EC), the funds for these independent clinical trials might come partly or entirely from public funds or charities  2  and the sponsor is generally the organisation that receives the funds.
WHO Trial Registration Data Set (TRDS) defines the primary sponsor as the individual, organisation, group or other legal entity which takes responsibility for initiating, managing and/or financing a study, and for properly registering the trial; it may or may not be the main funder. In IICTs, the sponsor has the legal capability to establish contracts with individuals (eg, principal investigator) or other stakeholders to assure the implementation and conduct of a clinical trial. When analysing all clinical trials in one database, during a specific timeframe, most of the trials testing medicines are mainly industry sponsored 3 with commercial interests. On the other hand, according to the European Medicines Agency, around 40% of trials registered in Europe are IICTs mainly sponsored by academia.
MethODs We identified past or current IICTs registered and starting in one European country (Portugal) from 1 January 2004 to 31 January 2017 in order to identify the funding source. Previously, we investigated trials that are ongoing in Portugal in only one database (EU Clinical Trials Register; EU-CTR). 5  In the current study, we extended the number of databases used and the type of intervention, considering the same country as a pilot. The selection of clinical trials registries (CTRs) for this study was based on trials registered and recruiting in Portugal. We first screened the WHO International Clinical Trials Registry Platform (WHO-ICTRP), a network platform that is directly updated by CTRs that comply with the required 24 quality-based items in the WHO-TRDS. This search enabled us to identify the presence of Portuguese trials in four of the main international registries: ClinicalTrials.
Step 2 The search in the EU-CTR was performed by year (from 1 January 2004 until 31 January 2017) with the terms 'Portugal and non-commercial' in the general search field. The duplicates found in each year were sequentially identified and not considered for the total number. Step 3 With the ClinicalTrials. gov registry, it was not possible to distinguish between commercial and non-commercial trials. Therefore, an advanced search was performed selecting the options 'Portugal' and 'Interventional studies', and was limited to the above-mentioned time period. Step 4 Registrations in the BiomedCentral CTR are associated with ISRCTN. In this CTR, it was not possible to refine the search besides selecting Portugal as the recruiting country. Trials initiated before or after the selected timeframe were discarded.
Data extraction Step 6 Information about trial identification (Trial ID) numbermain and secondary, recruitment status, sponsor name and country, and funding source-were extracted manually and independently from the registered records (from 1 February 2017 to 31 March 2017) and organised in Excel sheets. Forty per cent of the records were doublechecked by two of the authors (CM and FS). Sponsors were coded as being university, disease-specific organisation (disease associations or research institutes dedicated to a specific therapeutic area), hospital, research institute (non-specific therapeutic area), foundation and other (eg, international funding agency and private health clinic). Funders were coded as being (1) a non-profit organisation (eg, public institution, funding agency, disease-specific organisations), (2) a for-profit organisation (ie, private company/industry), (3) both, when funding was Open access provided by industry and non-profit organisation(s) and (4) not indicated, when the informationwas not provided.
Information on funder type is indicated in the field entitled 'sources of monetary support' in both WHO-ICTRP and EUCTR; in the 'sponsor and collaborators' field in ClinicalTrials. gov; and in the 'funder' field in ISRCTN and ANZCTR. When only the sponsor was mentioned in the ClinicalTrials. gov registry, we considered it as the funder. Additionally, in this registry, the support of funding agencies was perceived through secondary IDs where the code of the grant agreement is in some cases added. Exclusive funding for PhD or postdoctoral grants was not considered.
Identification of duplicates and complementary information Step 7 All trials identified as non-commercial were added to the same Excel sheet and organised by sponsor name. Duplicates were identified when secondary IDs were provided or when the sponsor and the title of the study were the same. Information obtained from different CTRs about Open access the same study was gathered in the final record (Step 9) and used for further analysis. Step 8 A closer look at each registration allowed the identification of more studies with no recruiting sites in Portugal as well as commercial trials. Step 9 After removing duplicates or discarded trials, the database with the final number of registrations was organised; information about funding was gathered when more than one registration was detected in Step 7.
search of publications of completed IICts Step 10 Publications from the completed IICTs were manually and independently screened. The Trial ID was used to search abstracts of journals indexed in Medline (using PubMed) as well as the four CTRs. Other publications were also found through Google search using the study title or the Trial ID. Using the name of the principal investigator, it was possible to find publications about the study, when no paper was found with the previous strategies. Surprisingly, in some publications, the Trial ID was not included in the abstract, which rendered the search more difficult. Step 11 Information about the funding source in each publication was manually and independently retrieved and compared with the information obtained from the databases by two of the authors (CM and FS).
Identification of non-commercial studies The number of trials identified in each database is shown in figure  1. After discarding industry-sponsored trials, non-interventional studies or those with no recruiting sites in Portugal, 601 non-commercial trials were considered eligible from all the screened databases. From those, 260 duplicates were identified, and 59 trials were discarded (figure  1 ). The main reason for the high number of discarded trials was that in the EU-CTR, the country is used as a general term. Therefore, registrations with Portugal as (1) a planned country but not yet approved by competent authorities (n=18), (2) the owner of the marketing authorisation or (3) the producer of the medicine (n=27) were also detected. As expected, a high number of trials was retrieved from the WHO-ICTRP (n=234, figure  1 , Step 1) as this platform receives weekly updates of the registrations in each of the CTRs.
Information available in each database In this study, we identified 282 IICTs among the nearly 3000 registrations in the four CTRs used and WHO-ICTRP. We had to manually screen the trials to distinguish between commercial trials and non-commercial trials, as this was not a systematic search option in three of them. The ability to filter this information was better with EU-CTR and ANZCTR. In ANZCTR, the search may be performed by sponsor type with eight categories; in EU-CTR, it is possible to consider the option of commercial or non-commercial study (figure  2A ). Since September 2017, it has been possible to search trials according to the funder type in ClinicalTrials. gov using the option 'All others', which includes non-profit organisations. Other differences in the information available in each database were ranked in terms of clearness according to our user experience.
The evaluation is pictured in figure  2A , which clearly shows the need for improvement in the information provided, specifically in regards to non-commercial status, funding source and secondary ID. Although ISCTRN and ANZCTR have a field where the name of the funder can be added, this information is not mandatory. However, when compared with the other databases, these two clearly distinguish the sponsor from the funder with two separate fields for this purpose, and in 95%-100% of registrations the name of the funder is different from the sponsor (table  1 ). Clini-calTrials. gov only displays one field named 'Sponsor and Open access collaborators' that is not always consistent with the information provided in the EU-CTR when the study is registered in both databases (Supplementary S. I. 1 table  1 ).
Collaborators were not indicated in 6% of registrations in ClinicalTrials. gov, and in 51% of cases (where they were indicated), the collaborator was different from the sponsor (table  1 ). In the other CTRs, a higher percentage of registries had the funder different from the sponsor (68%-100%). The EU-CTR has a field entitled 'Source(s) of monetary or material support for the clinical trial', which might be ambiguous regarding the real origin of the financial resources and ownership of the results. For example, when the name of a private company is included in this field, it is not possible to know if this company is providing either financial support, medicines, or both, and who will be the owner of the obtained results: the private company or the investigator. Additionally, in around 52% of registrations in this CTR, no information about the funder was provided (table  1 ).
The majority of the trials (n=235) were registered in only one CTR (figure  2B ), but when registered in different CTRs, the information about the trial was gathered in our records. Databases have a field to add a secondary ID but the information is not always provided (figure  2A ) and only 12 out of 47 trials registered in more than 1 CTR include the secondary ID (see Supplementary S. I. 1 table 2). The absence of secondary ID for the same trial gave rise to different records in the WHO-ICTRP (resulting in separate entries or duplicates). Only 28% of trials registered in more than one CTR (13 out of 47) had consistent information about funding (inset of figure  2B ) highest percentage (64%) of inconsistencies was detected when trials were registered in both ClinicalTrials. gov and other CTR(s) (table  1 ).
Funders of non-commercial trials in Portugal The analysis of sponsor organisation type showed that universities (n=139), disease-specific organisations (n=58) and hospitals (n=46) are the most frequent sponsors of non-commercial trials (figure  3A ). Industry was identified as the unique funding source in 33 trials, while 24 studies were funded both by industry and non-profit organisations. It was not possible to identify the funder in 23 registered trials and the sponsor was considered as the funder in 79 registrations (mainly from ClinicalTrials. gov). In the end, 123 trials received funds exclusively from non-profit organisations. A more detailed analysis showed that a total of 51 trials received monetary support from national and international funding agencies (figure  3B ). Among non-profit organisations, funding agencies are providing monetary support to 51 trials (23%), alone or together with other non-profit organisations (figure  3C ).
Consistency of information provided in databases and publications about funding of non-commercial trials In this study, we identified that a total of 133 trials, corresponding to 47% of the total number of trials, were completed (figure  4A ). Only 77 of these trials have already been published. If, however, we consider trials completed by 2015, the percentage of trials published increases to 60% (67 out of 111); this takes into account the 2 year, poststudy timeframe before publication. Information about the funding source was screened in each publication, showing that 78% (60 out of 77) of these publications referred to the source of funds (figure  4B ). Additionally, in 55% of the trials, the information collected from the databases was consistent with the information provided in publications (even when no funding was mentioned in both), whereas in 20% of the trials, the publication provided additional information.
DIsCussIOn In this study, a systematic analysis of IICTs registered in four CTRs and also in WHO-ICTRP was conducted with the main goal of understanding how these trials are funded in Portugal, used as a case study. The search was not straightforward and several incongruences were found after a laborious and time-consuming process of screening each of the 282 registered non-commercial clinical trials. Several IICTs (16%; n=47) were registered in different CTRs but secondary IDs were not always provided (Supplementary S. I. 1 table 2). Surprisingly, for the same study, information about funding was not consistent in 72% of cases (or 34 out of 47) when trials were registered in more than one CTR (figure  2B  and Supplementary table  1 ). Higher numbers of inconsistencies were found in registrations in ClinicalTrials.
gov when compared with the other CTRs (table  1 ). Even WHO-ICTRP, as previously indicated by others, does not reliably identify trials listed in multiple CTRs, making manual searches necessary. 7 8 e cannot explain the fact that some registrations in the screened CTRs were not detected in the WHO-ICTRP search, even though it was a broad search. Moreover, it is also not possible to explain why some registrations found in the CTRs were still not identified in the WHO-ICTRP when the ID number was searched directly (eg, NCT01280929, NCT01281098, NCT00189553, NCT00561054). Raftery et al showed the number of UK non-commercial trials registered by funding source from 1990 to 2009 and emphasised the fact that the trials registered in Clinicaltrials. gov had an unknown funding source. 9 ur findings are fully in line with these authors' views and herein we reiterate the need to improve the information about the funding source and the ownership of the results in Clinicaltrials.
gov. We believe that this is a way to increase the quality of records and the transparency of results of non-commercial trials. The inclusion of the funding source in Clinicaltrials. gov was one of the requirements for the Proposed Rule by Title VIII of the Food and Drug Administration Amendments Act of 2007  10  and it is mandatory in the TRDS for CTRs that provide information to the WHO-ICTRP. Nevertheless, to date, the clear identification of the funders and independence of results from commercial interests have not been considered as criteria when discussing the need for improvement of the available databases. Inconsistencies were also found when comparing information about funding sources in CTRs and publications of completed IICTs. Publications are not necessarily listed on trial registries, requiring an extensive manual effort to Make it a registration requirement in all databases to provide information about the (i) funding source, (ii) the role of the investigator and the funder in the design of the study and (iii) the ownership of the results When industry is the funder it is not possible to have information about the ownership of the results, unless the study is published With WHO-ICTRP some trials registered in different CTRs are not identified as duplicates  7 8  Improve this platform for detection of duplicates using not only the secondary ID but also the title of the study and the name of the sponsor  Open access identify them.
On the other hand, 20% (57 out of 282) of IICTs have received support from the industry (figure  3B ). Even though this support could be money or other type, when the study is funded or supported by a pharmaceutical company it is not clear who will be the owner of the obtained results. If the owner is the private company, the study should not be labelled as an IICT. Currently, through the screened databases, it is not possible to confirm if the ownership of the results belongs to the investigators when the funder is a pharmaceutical company. As suggested by others, public CTRs should ensure adequate monitoring of trial registrations; this would ensure completion of mandatory contact information fields identifying scientific leadership to promote accountability and transparency in clinical trial research.
11 In some cases, information about scientific leadershipis only clear when the results are published in scientific journals in which the ownership of the trial design and results is claimed by the clinical investigators involved in the study. 12 It would be important to add this information in the registration phase to ensure that questions over funding do not have a negative effect on the use of IICTfindings on the definition of new public health policies and implementation of new therapeutic approaches. In this study, a systematic analysis of the information provided by the sponsor about the funding source is presented for the first time in order to clearly identify IICTs. When searching the five databases, we faced several limitations and suggest the following strategies for improving the accuracy of clinical trial databases (table  2 ).