On Guidelines for Comparative Effectiveness Research Using Nonrandomized Studies in Secondary Data Sources

Generating knowledge on the comparative effectiveness of medical products with nonrandomized studies is challenging and will not infrequently result in biased effect estimates if epidemiologic principles are not followed [1]. Nevertheless, a recent study by Academy Health suggested that currently, about 85% of studies producing comparative effectiveness research (CER) are nonrandomized and I would not expect too much change in the near future as much as we all love randomized effectiveness trials [2]. The sizable and sudden influx of research funds to perform CER via the American Recovery and Reinvestment Act in the United States is a unique chance to help jump-start and establish the field. This challenge comes with a nontrivial risk. What if after all the money is spent, it turns out that a good proportion of the studies produced invalid results or results that will not answer the questions that decision-makers struggle with? The nonrandomized world of research on the effectiveness of therapeutics is complex, certainly more complex than the world of randomized trials. The now published guidelines on “Good Research Practices for Comparative Effectiveness Research with Nonrandomized Studies of Treatment Effects using Secondary Data Sources” [3‐5] by the corresponding ISPOR Task Force will serve as a welcome reminder that we should keep the basics in mind when embarking in CER: be clear on what questions we want to answer and make sure we answer it in a way that policymakers and physicians can meaningfully act upon the new evidence. Start out with an incidence user design, define an exposure risk window that is appropriate given a product’s biological action, thoughtfully choose a comparison group with identical or similar indication and of practical relevance, and pay utmost attention to confounding by indication caused by the channeling of patients into treatment groups based on their prognosis related to the study outcome [6]. These considerations paired with solid clinical knowledge and understanding of realworld utilization will avoid many grave mistakes. Most of the advanced analytic methods discussed in the guidelines are important additions to the epidemiologist’s toolbox but will not overcome errors in fundamental design consideration. How are guidelines different from pharmacoepidemiology textbooks? They might be more up-to-date but less detailed. A major similarity, however, is that one will have to read guidelines, learn more about specific issues depending on one’s background, and study new methods. If researchers are not willing to go back to school with these guidelines, their mission is largely in vain. Thisraisesafundamentalissuethatiscriticalfortheestablishment of a functioning and productive CER community: the capacity to produce researchers trained in the advanced methods of pharmacoepidemiology and technology assessment using nonrandomized studies. Such training is available in some but not many academic institutions in North America and Europe, but not enough graduates are produced to satisfy the growing demand. The National InstitutesofHealthdoesnothaveaT32traininggrantmechanism for our discipline, which seems to fall between the cracks of an organization largely organized by disease areas. Rigorous training is the most effective way to ensure a high quality of nonrandomized research on the effectiveness of medical products. The authors of these guidelines are to be applauded for their work, and now, it is up to the practitioners to study them and brush up on some newer methodologies, while the guidelines can go into round 2 of an ongoing revision cycle. Disclosure: Dr. Schneeweiss is the president of the International Society for Pharmacoepidemiology and is the principal investigator of the Brigham and Women’s Hospital DEcIDE Center for Comparative Effectiveness Research funded by the Agency for HealthCare Quality and Research. The opinions stated in the article are his own and not representing any of those organizations.