Comment on: "Zoo or savannah? Choice of training ground for evidence-based pharmacovigilance".

We read the article by Noren et al. [1] with great interest and commend their effort in bringing forward a critical issue in the evaluation of signal detection methodologies, namely the choice of a benchmark (a reference standard) and an associated evaluation strategy. Noren et al. argue that signal detection is fundamentally a prognostic activity. Therefore, evaluation strategies should aim to emulate a prospective analysis of signal detection in lieu of a commonly applied yet unsatisfactory approach of retrospective analysis based on well established associations such as those comprising the Observational Medical Outcome Partnership (OMOP) [2] and EUADR benchmarks [3]. Noren et al. demonstrate that the two evaluation strategies may lead to different conclusions. They partially attribute this discrepancy to biasing effects (e.g., the influence of publicity on spontaneous reporting and on patient management), which are a consequence of examining well established associations in a retrospective manner. Taken together, Noren et al. argue that evaluations should be based on benchmarks consisting of emerging or recently labeled adverse drug reactions (ADRs), which are to be applied in a manner that simulates prospective analysis by backdating the analyses to periods prior to the conception of these ADRs. We agree with the issues raised by Noren et al., but do not go as far as the dismissal of existing benchmarks. In an effort to shed light over this debate we recently created a time-indexed benchmark specifically designed to support the type of prospective evaluations proposed by Noren et al. The benchmark consists of recently labeled adverse events communicated by the US FDA in 2013. It includes 62 positive controls and 75 negative controls, covering 38 adverse events and 44 drug ingredients. Together with its description, the benchmark is available through Nature Scientific Data [4]. A preliminary investigation that applied this benchmark to evaluate FDA Adverse Event Reporting System (FAERS)-based signal detection provides support for the argument by Noren et al., in contrast with our earlier study based on the OMOP benchmark [5]. Despite these results we maintain our view that the two approaches should supplement each other. A key advantage to using well established positive controls is in the reliability of their supporting evidence. In a benchmark created prior to the inception of a given recently labeled or emerging ADR, it is possible that this ‘true’ ADR would have been classified as a negative control. Likewise, the status of a recently labeled ADR (positive control in some benchmarks) may be revised based on new refuting evidence. Thus, the increased level of uncertainty associated with experiments based on such recently labeled or emerging ADRs cannot be ignored. Another issue is that many post-approval adverse events emerge shortly after a drug is introduced to the market. This short duration suggests that a backdated prospective analysis of benchmarks containing newly introduced drugs (an important target for monitoring) may not be feasible given that an insufficient amount of data will be available for analysis. In such cases, a retrospective analysis is likely the only option. Perhaps the most important issue is the interpretation of backdated analyses. A key question that follows a backdated analysis is whether or not the conclusion drawn from R. Harpaz (&) W. DuMouchel Oracle Health Sciences, Bedford, MA, USA e-mail: rave.harpaz@oracle.com