dataesr/scientific-paragraphs-categorization huggingface dataset:
This finding has been validated excluding ADRs identified prior to authorisation from the reference database but further work is ongoing to characterise this effect  [35]  Recommendation Rationale Consideration should be given to carrying out comparisons of quantitative signal detection methods across spontaneous report databases matching at the drug-event combination level rather than averaging over all drug-event combinations It is possible that some ADRs may be more easily found in some databases. Other published studies have suggested some benefits of stratified and subgroup analyses but often the analyses included only a few key covariates or study products, and were conducted in single databases  [36] [37] [38] [39] [40] [41] [42] [43] [44]. It is not clear how generalisable these results are to other spontaneous report datasets of different sizes and characteristics.
Additionally, to our knowledge a head-to-head comparison of stratified and subgroup analyses against a reference standard has not been conducted. The study of Seabroke et al. [45]  investigated the impact of stratification and subgrouping in signal detection algorithms in spontaneous report databases of different sizes and characteristics using a range of key covariates (age, gender, calendar time period, country of origin, vaccines/non-vaccines, event seriousness, reporter type and report source). Signal detection performance was measured against a reference standard. Disproportionality analyses were conducted using either stratified or subgroup approaches and compared with an unstratified crude analysis. Stratified and subgroup analyses calculated disproportionality statistics within each individual stratum separately.
For the stratified analyses, these were combined into a single value using a Mantel-Haenszel approach, whereas for subgroup analyses, a positive signal was counted if any of the individual strata met the signal criteria. The results were presented as sensitivity and precision (positive predictive value) for each approach calculated using a reference set of ADRs compiled from the SPC and company core data sheets as a proxy for true positives. Additional analyses included investigating the benefit of combined subgroup/stratified variables and also investigating the impact of a permutation analysis that used randomly split strata of equal size to a real variable of interest and compared the results with those for the real variable. Whilst the spontaneous report databases employed in this study included large international, national and industry datasets, the results may not be generalisable to all spontaneous report databases, particularly those with a small volume of reports.
The results from this study showed that subgroup analyses consistently performed better than stratified analyses in all databases. Subgroup analyses were also shown to provide clear benefits over crude analyses for some databases whilst stratified analyses were not found to increase either sensitivity or precision beyond that associated with analytical artefacts of the stratified analysis. For the smaller spontaneous report databases, a gain in precision tended to result in some loss of sensitivity particularly for the stricter disproportionality method/ threshold and for the regulatory dataset with reports from only one country  [45]  Subgrouping by seriousness of ADR or routinely excluding legal cases is unlikely to provide benefits in signal detection in terms of increased sensitivity or precision Subgrouping by seriousness of the ADR defined using the IME list a had little effect on sensitivity or precision in any spontaneous report database.
An analysis excluding cases submitted by lawyers also had little effect in all spontaneous report databases apart from the largest international database, which showed an increase in sensitivity and precision when legal cases are excluded  [45]  Subgrouping by gender, reporter type and 5-yearly time points may provide modest improvement in precision in all, and sensitivity in some, spontaneous report databases Subgrouping by gender, reporter type and 5-yearly time points showed a modest improvement in precision for all spontaneous report databases and improved sensitivity for larger and international databases. Implementation of these subgroup analyses into routine signal detection may provide some benefit  [45]  2.
6 Influence of Masking on Disproportionality Analysis Disproportionality analysis uses the spontaneous report database itself as the basis for computing an expected number of reports on a particular drug and adverse event. This is based on the assumption that true causal relationships between drugs and events do not influence the overall reporting rates and that the degree of under-(or over-) reporting for a given event is approximately the same for all drugs. In practice, this assumption may be invalid for some drugs and events in a spontaneous reporting database. For example, attention to a real or perceived safety issue in the medical community or in public media may increase the reporting rate for that drug-event pair to such an extent that the overall reporting of that event is affected and potentially masks alerts from other drugs  [46] [47] [48] [49] [50] [51].
The two studies that made up this PROTECT work package  [49, 50]  looked at the impact of masking on disproportionality analysis. An algorithm was used to generate a masking ratio whereby the masking effect of one drug on another can be estimated for a specific event and suggests a simplified version valid under certain conditions. A simulation study was performed that focussed specifically on comparing differences between the simplified and exact masking ratio  [49]. The impact of different approaches to treating the reports was also explored, including both the suspected masking drug and the suspected masked drug as concomitant medicines. In a follow-on study, the extent and impact of masking was studied in two spontaneous reporting data sets: EudraVigilance and Pfizer's spontaneous report database  [50].
Recommendation Rationale Quantification of the masking effect of drugs on adverse reactions or adverse reactions on drugs could be used as a diagnostic tool of the extent of masking at two levels: For determining, based on the general characteristics of a spontaneous report database, if the application of an unmasking algorithm would be worthwhile At the product-event pair level, if a specific concern is raised about a potential masking effect driven by another product or another group of products Results indicate that many drugs and adverse reactions are not affected by masking. Avoid complicating the analysis of data by adding an unmasking procedure when masking is not an issue. Formulas for assessing the effect of masking can be found in papers by Maignen et al. [49, 50]. As the characteristics of the spontaneous report database change over time, it is still interesting to monitor the extent of masking periodically.
Duplicate Detection Individual case reports of suspected harm from medicines are fundamental for signal detection in post-marketing surveillance. Their effective analysis requires reliable data and one challenge is report duplication. These occur when multiple unlinked records are recorded describing the same suspected ADR in a particular patient. Report duplication is known to occur for a diverse range of reasons, including reporting of an ADR from multiple primary sources, requirements for marketing authorisation holder reporting of literature cases (and subsequent retransmission), and technical or administrative issues  [56]. De-duplication of data is also widely understood to be a time-consuming process; however, little research had been undertaken prior to PROTECT on the efficiency of different de-duplication methodologies.
Duplicate ICSRs are known to distort statistical screening  [56]  both increasing the numbers of falsepositive and false-negative signals. The net effect of duplicates (and de-duplication techniques) is unknown, and may not be consistent across all products in the database; however, where there is a significant impact there is potential for the cases to mislead clinical assessment. Many organisations rely on rule-based duplicate detection methods, which rely on exact matching of a number of elements within individual case safety reports and displaying the putative duplicate cases for review. Rule-based approaches can vary significantly in their complexity and based on the fields used, and may consider patient, reporter, drug or reaction details, or a combination of them. The study compared methods  [57]  used in the UK, Denmark and Spain.
The UMC had previously published a probabilistic record matching algorithm that indicates the likelihood of cases being duplicates (vigiMatch  [58] ) as an alternative to rule-based approaches. The PROTECT study attempted to quantify the benefits (or disbenefits) of the different approaches used by PROTECT consortium members. A first phase of the study aimed to evaluate probabilistic record matching for duplicate detection compared with rule-based approaches. Studies considered positive predictive value and numbers of false positives of different approaches, and in addition attempted to characterise the main causes of duplication. Initial research was undertaken using the WHO Global Individual Case Safety Reports Database, VigiBase, for reports submitted between 2000 and 2010. Suspected duplicates for the UK, Denmark, and Spain were reviewed and classified by the respective national centre.
This included evaluation to determine whether confirmed duplicates had already been identified by in-house rule-based screening. A second phase of the study directly compared results from the MHRA's rule based approach to the UMC's probabilistic record matching approach using data received in the MHRA's Sentinel database during 2013. Probabilistic record matching performed positively when compared with rule-based approaches. Specifically, probabilistic record matching demonstrated a high predictive value above that of rule-based methods and is expected to improve efficiency and accuracy of duplicate management. The study showed very few false positives suggesting it may be possible to increase sensitivity while ensuring false-positive rates are kept at a reasonable level. The study  [57]  highlighted that case management system changes or upgrades can occasionally result in large numbers of duplicates, either in the source system, or those external to the organisation.
It was also noted that proliferation of duplicated cases within databases occurred as a result of rapid submission and re-transmission of cases to multiple stakeholders made possible by electronic systems. This emphasises the need for swift and robust duplicate detection procedures at each organisation. Data privacy requirements were noted as a barrier to the most effective duplicate detection and it was considered that approaches to deidentifying data (for example, scrambling dates and patient initials) in a way that permits duplicate detection should be pursued to allow for effective duplicate detection in databases that pool reports from different sources. Although beyond the scope of the PROTECT study, further evaluation should be undertaken to understand the feasibility and impact of automatic exclusion of potential duplicates from quantitative signal detection algorithms.
Hence, it makes sense to ask whether there is a direct relationship between the magnitude of disproportionality statistics and the magnitude of the association between a product and an adverse effect from pharmacoepidemiological studies. A study was conducted to determine the proportional reporting ratios (PRRs) for a set of known ADRs and compare them with estimates of association from formal epidemiological studies  [59]. A set of 15 confirmed ADRs were selected from the initially identified dataset of pharmacovigilance driven European Union regulatory actions and for which relative risk estimates from formal studies were available and were considered to provide Further work should be undertaken to explore lowering the threshold for the tested probabilistic record matching method and methods in general to evaluate the balance of false positives and negatives Our study showed very few false positives, so it should be possible to increase sensitivity while ensuring false positive rates are kept at a reasonable level  [57]  Recommendation Rationale Further evaluation should be done to understand the impact of automatic exclusion of potential duplicates from quantitative signal detection algorithms This was beyond the scope of the PROTECT study.
If this approach proved successful manual duplicate detection activities could be eliminated resulting in time/resource savings  [57]  Approaches to deidentifiying data (for example, scrambling dates and patient initials) in a way that permits duplicate detection should be pursued to allow for effective duplicate detection in databases that pool reports from different sources This will reduce the negative impact of data privacy laws, for duplicate detection in international databases  [57]  well-established evidence supporting the respective regulatory actions. Prior to any calculation of PRR, the studies were collated and a best estimate of the risk ratio selected on the basis of pre-specified rules. When available, this estimate was that determined during the European Union regulatory assessment.
At the same time, an estimate was made of the date at which each ADR was first publicly recognised. Only after the risk ratio was decided upon was the PRR calculated by reconstruction of the spontaneous reporting data system at the predetermined date of first recognition in the medical community of the ADR. The primary analysis used the EudraVigilance spontaneous report database and an additional analysis was carried out in FEDRA, the Spanish national spontaneous report database. Case definitions for the ADR of interest were developed for PRR calculations for each drug-event pair, aiming to reproduce the case definition as used in the epidemiological studies providing relative risks. An initial dataset of 78 drug-event pairs was obtained following the initial inclusion criteria. Following the exclusion criteria, 15 drug-event pairs were finally selected.
All analyses were performed in The Health Improvement Network (THIN 4  ) database of longitudinal electronic health records from general practices in the UK. The comparison between exploratory and confirmatory analysis was based on 13 published studies of possible ADRs in THIN  [71]. The selected studies listed a total of 56 drugevent pairs that were included in the analysis. For each pair, the analysis results of the study (positive or negative) were compared with those that would result from analysis with standard design choices for a calibrated self-controlled cohort analysis in THIN. Observed differences were closely scrutinised to identify areas of possible improvement for the standard implementation of the design, for exploratory analysis. In our comparison to published epidemiological studies, a common discrepancy was that the epidemiological study performed analysis for all drugs in a class together and/ or for a number of related medical events together, which improves power.
The literature is very sparse on such comparisons. One example is from Trifiroet al. [73] , who as part of the EU-ADR (European Commission sponsored project 'Exploring and Understanding Adverse Drug Reactions by integrative mining of clinical records and biomedical knowledge'), looked at the number of drug-event combinations that were highlighted as disproportional in each of the data sources. Analyses were performed in THIN and VigiBase, each backdated to the end of 2004, to match approximately the time of the historical signals. The analysis in VigiBase was repeated with a restriction to reports originating only from the UK. The retrospective evaluation against historical safety signals for European centrally authorised products showed that none of them could be detected in THIN with the method we used, prior to the initial signal at the EMA.
In our evaluation of historical safety signals from the EMA, none of the positive controls could be detected in the THIN database at an early stage, whereas this was possible in VigiBase for some of the signals, even when we considered only the subset of the UK individual case reports  [72]  Recommendation In our comparison to published epidemiological studies, a common discrepancy was that they performed analysis for all drugs in a class together and/or for a number of related medical events together, which improves power. However, our detailed review often found substantial and important differences among different drugs in the same class or among different medical events in the same category.
Signal Detection in Clinical Study Data In the pre-approval phase of a drug, systematically collected adverse event data from randomised clinical trials are the principal data source and therefore the cornerstone of safety analysis. Specifically, clinical trial data allow the estimation of exposure, which is one of the limitations of spontaneous reporting datasets  [7]. In addition, randomisation itself is a powerful feature of randomised controlled trials that addresses the issue of confounding, which otherwise complicates attempts to identify imbalances in the incidence of adverse events between the treated and untreated subjects in data from non-randomised sources. Because of this, clinical trial data form a natural source for signal detection, especially in the early phase of a drug's lifecycle.
Two different types of analyses performed in company clinical trial databases are presented: in the first, the use of extreme value modelling for the prediction of potential drug toxicity was evaluated; in the second, different approaches for dealing with multiplicity issues for adverse event-based signal detection were compared. It is often the case that potential drug toxicity is suggested by the occurrence of extreme values of clinical variables, rather than changes in the location of the distribution of these variables. For example, large values of serum creatinine and blood urea nitrogen are used in the diagnosis of acute renal failure, large values of the QT interval are suggestive of cardiotoxicity and large values of alanine aminotransferase suggest hepatotoxicity. Many other examples of abnormally large (or small) values of laboratory variables being indicative of safety issues can be found in the Common Terminology Criteria guidelines,  5 and, in general, any laboratory values that fall above the upper limit of normal or beneath the lower limit of normal may be indicative of a potential safety issue.
One important characteristic of signal detection in clinical trials as well as in other databases is a focus on the evaluation of a large number of endpoints. Analyses of adverse event data typically generates multiple risk measurement estimates associated with events across several body systems. Selection of outcomes for further evaluation by conventional hypothesis testing of between-group differences for each endpoint can be problematic. Ignoring the fact that the data determine the hypotheses that are tested, by using a process that involves a large number of comparisons across multiple analysis time points multiplicity becomes an issue. Acknowledging P-values as a useful flagging device, ICH E9 recommends statistical adjustments for multiplicity when applying hypothesis tests to a large number of safety variables in clinical trial data.
However, probably owing to the concern of missing true safety signals (i. e. false-negative signals), multiplicity adjustment has also been described as counterproductive and again probably because of these and similar reservations, adjustments are often avoided, despite the fact that ignoring multiplicity may easily result in an excessive rate of false-positive signals. The EudraVigilance Expert Working Group notes that thresholds commonly used to detect signals in spontaneous data are a trade-off between two conflicting goals: ''either generating too many false positive signals if the threshold is too low or missing true signals if this threshold is too high''. Given this trade-off, it is important to identify and calibrate methods to strike a reasonable balance between these two parameters.
Discussion These recommendations are presented to highlight the outcomes of the research conducted under the auspices of IMI PROTECT Work Package 3 and in such a way that pharmacovigilance professionals, particularly those with an interest in research and methods development can readily adopt appropriate potential improvements in their quantitative signal detection practices. They should not be considered a comprehensive treatise on the subject of quantitative signal detection but should be considered by The use of Bayesian Hierarchical Models can improve the efficiency of signal detection through borrowing of strength from other relevant events in the clinical trial dataset. This must be weighed against the more complex computational requirements of Bayesian modelling Bayesian Hierarchical Models provided the best performance with regard to positive predictive value, specificity, sensitivity and negative predictive value, mainly owing to their ability to ''borrow strength'' across similar terms  [75]  The use of more specific MedDRA Ò groupings can further improve signal detection in clinical trial data The use of narrow-term groupings for analysis provided slightly better results for signal detection compared with the analysis based on MedDRA Ò PTs alone  [75]  readers within the context of the entire body of signal detection research and guidance documents that exist on good signal detection practice  [7, 77]  outside of IMI PROTECT.
For example, PROTECT's focus on quantitative signal detection methods means that recommendations relating to the relative merits of quantitative vs. qualitative signal detection methods cannot be made and pharmacovigilance organisations are likely to have to continue with both approaches to maintain an optimal signal detection capability. Although a broad cross-section of databases  [78]  and data sources were employed in the work package, the generalisability of recommendations to other databases and data sources should be considered carefully before implementing them, particularly in databases of adverse event reports smaller than those used in this project. An important strength of PROTECT's signal detection research was the execution of standardised analysis protocols across multiple spontaneous reporting datasets  [35, 45].
Several of these studies compared the databases of pharmaceutical companies, national regulatory authorities, and international organisations, such as the EMAgency and the WHO. PROTECT found no overall benefit in conducting signal detection using MedDRA Ò HLTs or SMQs compared with using individual PTs  [9]. Some relatively minor gain in time to signalling was seen when closely related (in a clinical sense) ADR terms were grouped together, an area for potential future research. This is compatible with an earlier study of the US FDA Adverse Event Reporting System, which found that analysis at the level of HLTs or SMQs decreased sensitivity but increased specificity. Its reference dataset were drug-event combinations with some degree of support in the literature that were not on the original drug label, but it did not consider timeliness  [8].