Abstract: Seven studies met the inclusion criteria. No studies had investigated selective inclusion of results in systematic reviews, or discrepancies in outcomes and analyses between systematic review registry entries and published systematic reviews. Based on a meta-analysis of four studies (including 485 Cochrane Reviews), 38% (95% confidence interval (CI) 23% to 54%) of systematic reviews added, omitted, upgraded or downgraded at least one outcome between the protocol and published systematic review. The association between statistical significance and discrepant outcome reporting between protocol and published systematic review was uncertain. The meta-analytic estimate suggested an increased risk of adding or upgrading (i.e. changing a secondary outcome to primary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.43, 95% CI 0.71 to 2.85; two studies, n = 552 meta-analyses). Also, the meta-analytic estimate suggested an increased risk of downgrading (i.e. changing a primary outcome to secondary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.26, 95% CI 0.60 to 2.62; two studies, n = 484 meta-analyses). None of the included studies had investigated whether the association between statistical significance and adding, upgrading or downgrading of outcomes was modified by the type of comparison, direction of effect or type of outcome; or whether there is an association between direction of the effect estimate and discrepant outcome reporting. Several secondary outcomes were reported in the included studies. Two studies found that reasons for discrepant outcome reporting were infrequently reported in published systematic reviews (6% in one study and 22% in the other). One study (including 62 Cochrane Reviews) found that 32% (95% CI 21% to 45%) of systematic reviews did not report all primary outcomes in the abstract. Another study (including 64 Cochrane and 118 non-Cochrane reviews) found that statistically significant primary outcomes were more likely to be completely reported in the systematic review abstract than non-significant primary outcomes (RR 2.66, 95% CI 1.81 to 3.90). None of the studies included systematic reviews published after 2009 when reporting standards for systematic reviews (Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement, and Methodological Expectations of Cochrane Intervention Reviews (MECIR)) were disseminated, so the results might not be generalisable to more recent systematic reviews. Discrepant outcome reporting between the protocol and published systematic review is fairly common, although the association between statistical significance and discrepant outcome reporting is uncertain. Complete reporting of outcomes in systematic review abstracts is associated with statistical significance of the results for those outcomes. Systematic review outcomes and analysis plans should be specified prior to seeing the results of included studies to minimise post-hoc decisions that may be based on the observed results. Modifications that occur once the review has commenced, along with their justification, should be clearly reported. Effect estimates and CIs should be reported for all systematic review outcomes regardless of the results. The lack of research on selective inclusion of results in systematic reviews needs to be addressed and studies that avoid the methodological weaknesses of existing research are also needed.

Plain-language summary: A systematic review summarises evidence from multiple studies to answer a specific research question (e.g. what are the benefits and harms of a particular intervention for a particular health condition?). Often, there are many outcomes that systematic review authors could report to address their research question (e.g. pain, disability and quality of life for patients with musculoskeletal conditions) and many different results available for a particular outcome (e.g. a study might measure pain using three different scales at four time points). If the decision about which outcomes to investigate in a systematic review is made based on the results for those outcomes in the eligible studies, this may lead to bias. While, if the decision about which outcomes to report in a systematic review and the ways to report them is based on the results, this may mislead users of the systematic review. This methodology review summarises the findings of studies examining the inclusion of results and reporting of outcomes in systematic reviews. We searched for studies indexed in electronic bibliographic databases up to May 2013. We included seven studies and found that outcomes investigated and reported in systematic reviews were often changed between the protocol and published systematic review. We also found that it was unclear whether the decision to make these changes was related to how statistically convincing the treatment effect for that outcome was. More studies are needed to confirm if this relationship exists. Also, one study found that some systematic reviews did not report all of the most important outcomes in the abstract of the review. Another study found that outcomes with a more statistically convincing result were more likely to be completely reported in the abstract than other outcomes. The studies that we included were limited to systematic reviews published before 2009. New studies are needed to examine the inclusion of results and reporting of outcomes in more recent systematic reviews.