# A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism

Céline Chapelle, Gwénaël Le Teuff, Paul Jacques Zufferey, Silvy Laporte, Edouard Ollier

<figure><img src="image_1.png" /></figure>

## To cite this version:

Céline Chapelle, Gwénaël Le Teuff, Paul Jacques Zufferey, Silvy Laporte, Edouard Ollier. A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism. Research Synthesis Methods, 2023, 15 (1), pp.117 - 129. 10.1002/jrsm.1676. inserm-04494324

HAL Id: inserm-04494324

https://inserm.hal.science/inserm-04494324

Submitted on 7 Mar 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism

Céline Chapelle, Gwénaël Le Teuff, Paul Jacques Zufferey, Silvy Laporte, Edouard Ollier

<figure><img src="image_1.png" /></figure>

## To cite this version:

Céline Chapelle, Gwénaël Le Teuff, Paul Jacques Zufferey, Silvy Laporte, Edouard Ollier. A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism. Research Synthesis Methods, 2023, 15 (1), pp.117 - 129. 10.1002/jrsm.1676. inserm-04494324

HAL Id: inserm-04494324

https://inserm.hal.science/inserm-04494324v1

Submitted on 7 Mar 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism

Céline Chapelle<fnref n="1" /> ID

Gwénaël Le Teuff<fnref n="2" />

Paul Jacques Zufferey<fnref n="3" />

Silvy Laporte<fnref n="1" />

Edouard Ollier<fnref n="1" /> ID

## Correspondence

Céline Chapelle, Service de pharmacologie clinique, CHU Saint-Étienne, Hôpital Nord, F-42055 Saint-Étienne Cedex 02, France.
Email: celine.chapelle@chu-st-etienne.fr

## Abstract

The number of meta-analyses of aggregate data has dramatically increased due to the facility of obtaining data from publications and the development of free, easy-to-use, and specialised statistical software. Even when meta-analyses include the same studies, their results may vary owing to different methodological choices. Assessment of the replication of meta-analysis provides an example of the variation of effect 'naturally' observed between multiple research projects. Reproducibility of results has mostly been reported using graphical descriptive representations. A quantitative analysis of such results would enable (i) breakdown of the total observed variability with quantification of the variability generated by the replication process and (ii) identification of which variables account for this variability, such as methodological quality or the statistical analysis procedures used. These variables might explain systematic mean differences between results and dispersion of the results. To quantitatively characterise the reproducibility of meta-analysis results, a bivariate linear mixed-effects model was developed to simulate both mean results and their corresponding uncertainty. Results were assigned to several replication groups, those assessing the same studies, outcomes, treatment indication and comparisons classified in the same replication group. A nested random effect structure was used to break down the total variability within each replication group and between these groups to enable calculation of an intragroup correlation coefficient and quantification of reproducibility. Determinants of variability were investigated by modelling both mean and variance parameters using covariates. The proposed model was applied to the example of meta-analyses evaluating direct oral anticoagulants in the acute treatment of venous thromboembolism.

## KEYWORDS

aggregate data, Bayesian analysis, bivariate mixed effect model, replication, reproducibility, vibration of effects

<footnote n="1">¹Université Jean-Monnet, Mines Saint-Étienne, INSERM, U1059, SAINBIOSE, F-42023; Service de pharmacologie clinique, CHU Saint-Étienne, F-42055 Saint-Étienne, France, Université Jean Monnet, Saint-Étienne, France</footnote>

<footnote n="2">²Service de Biostatistique et d'Épidémiologie, Gustave Roussy, Université Paris-Saclay, Villejuif, France; Oncostat U1018, Inserm, Équipe Labellisée Ligue Contre le Cancer, Université Paris-Saclay, Villejuif, France</footnote>

<footnote n="3">³Département d'Anesthésie-Réanimation, Service de pharmacologie clinique, CHU Saint-Étienne, F-42055 Saint-Étienne; Université Jean-Monnet, Mines Saint-Étienne, INSERM, U1059, SAINBIOSE, F-42023, CHU Saint-Étienne, Saint-Étienne, France</footnote>
# A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism

Céline Chapelle<fnref n="1" /> ID

Gwénaël Le Teuff<fnref n="2" />

Paul Jacques Zufferey<fnref n="3" />

Silvy Laporte<fnref n="1" />

Edouard Ollier<fnref n="1" /> ID

## Correspondence

Céline Chapelle, Service de pharmacologie clinique, CHU Saint-Étienne, Hôpital Nord, F-42055 Saint-Étienne Cedex 02, France.
Email: celine.chapelle@chu-st-etienne.fr

## Abstract

The number of meta-analyses of aggregate data has dramatically increased due to the facility of obtaining data from publications and the development of free, easy-to-use, and specialised statistical software. Even when meta-analyses include the same studies, their results may vary owing to different methodological choices. Assessment of the replication of meta-analysis provides an example of the variation of effect 'naturally' observed between multiple research projects. Reproducibility of results has mostly been reported using graphical descriptive representations. A quantitative analysis of such results would enable (i) breakdown of the total observed variability with quantification of the variability generated by the replication process and (ii) identification of which variables account for this variability, such as methodological quality or the statistical analysis procedures used. These variables might explain systematic mean differences between results and dispersion of the results. To quantitatively characterise the reproducibility of meta-analysis results, a bivariate linear mixed-effects model was developed to simulate both mean results and their corresponding uncertainty. Results were assigned to several replication groups, those assessing the same studies, outcomes, treatment indication and comparisons classified in the same replication group. A nested random effect structure was used to break down the total variability within each replication group and between these groups to enable calculation of an intragroup correlation coefficient and quantification of reproducibility. Determinants of variability were investigated by modelling both mean and variance parameters using covariates. The proposed model was applied to the example of meta-analyses evaluating direct oral anticoagulants in the acute treatment of venous thromboembolism.

## KEYWORDS

aggregate data, Bayesian analysis, bivariate mixed effect model, replication, reproducibility, vibration of effects

<footnote n="1">¹Université Jean-Monnet, Mines Saint-Étienne, INSERM, U1059, SAINBIOSE, F-42023; Service de pharmacologie clinique, CHU Saint-Étienne, F-42055 Saint-Étienne, France, Université Jean Monnet, Saint-Étienne, France</footnote>

<footnote n="2">²Service de Biostatistique et d'Épidémiologie, Gustave Roussy, Université Paris-Saclay, Villejuif, France; Oncostat U1018, Inserm, Équipe Labellisée Ligue Contre le Cancer, Université Paris-Saclay, Villejuif, France</footnote>

<footnote n="3">³Département d'Anesthésie-Réanimation, Service de pharmacologie clinique, CHU Saint-Étienne, F-42055 Saint-Étienne; Université Jean-Monnet, Mines Saint-Étienne, INSERM, U1059, SAINBIOSE, F-42023, CHU Saint-Étienne, Saint-Étienne, France</footnote>
# Highlights

## What is already known

* Meta-analyses have recently proliferated to a massive extent, sometimes including the same studies and analysing the same outcomes, this phenomenon being designated result replication.

* Despite inclusion of the same studies, the results of meta-analyses may vary according to the methodological choices made.

* The reproducibility of meta-analysis results has mainly been explored using graphical descriptive representations with no quantitative analysis proposed.

## What is new

* In this study, a bivariate linear mixed-effect model was developed to simulate both mean meta-analysis results and their standard errors.

* A nested random effect structure was used to break down the total variability within each replication group and between groups, each replication group containing results assessing the same studies, outcomes, treatment indication and comparisons. Reproducibility was then assessed using an intragroup correlation coefficient. Covariate effects were included to characterise the variation of both mean and variance model parameters.

# 1 | INTRODUCTION

The current race to publish has led to a substantial increase in scientific articles published each year and raises issues in terms of scientific integrity and value. Owing to their high level of evidence, meta-analyses have become common practice in clinical research, and are therefore not exempt from this rush to publish.¹,² In particular, the number of meta-analyses of aggregate data has dramatically increased owing to the ease of extracting data from publications and the development of free, user-friendly and dedicated statistical software. This phenomenon has resulted in the replication of meta-analyses, with several analyses having the same objectives and including the same studies. It is often described in the literature as redundancy and has already been observed in many fields of medical research.³⁻¹⁶

Although investigating the same outcomes and including the same studies, these meta-analyses may differ in terms of the population of interest or the statistical methodology used, and these differences may influence their results. This phenomenon may be regarded as a ‘vibration of effects’ (VoE) and describes the extent to which the results of analyses could change according to the characteristics of the population included or the methodological choices made.¹⁷

The VoE phenomenon is classically investigated in a single research project by artificially producing multiple

meta-analyses from available studies.¹⁸,¹⁹ Assessment of the replication of meta-analysis results provides an example of VoE ‘naturally’ observed in multiple research projects. So far, the variability of meta-analysis results has mostly been reported using graphical descriptive representations,¹⁹⁻²¹ indicating the magnitude of effects (i.e., relative risk, RR) on the x-axis as a function of its level of significance (i.e., its *p*-value) on the y-axis. To the best of our knowledge, no quantitative analysis has been proposed up to now. The quantitative analysis of these results could enable breakdown of the total observed variability and subsequently quantification of the variability generated by the replication process. It could also allow identification of parameters likely to explain this variability, such as methodological quality or the statistical analysis procedures used. These parameters might explain systematic mean differences between meta-analysis results and/or dispersion (variance) of these results. Such quantitative analysis is thus essential to analyse the distortion (or improvement) in treatment effect estimates created by the phenomenon of replication and to better characterise the robustness of these estimates.

This paper aims to provide a general methodological framework for quantitative analysis of the reproducibility of meta-analysis results. The evaluation of direct oral anticoagulants (DOACs) in the acute treatment of venous thromboembolism (VTE) was chosen as an illustrative example of this approach.
# Highlights

## What is already known

* Meta-analyses have recently proliferated to a massive extent, sometimes including the same studies and analysing the same outcomes, this phenomenon being designated result replication.

* Despite inclusion of the same studies, the results of meta-analyses may vary according to the methodological choices made.

* The reproducibility of meta-analysis results has mainly been explored using graphical descriptive representations with no quantitative analysis proposed.

## What is new

* In this study, a bivariate linear mixed-effect model was developed to simulate both mean meta-analysis results and their standard errors.

* A nested random effect structure was used to break down the total variability within each replication group and between groups, each replication group containing results assessing the same studies, outcomes, treatment indication and comparisons. Reproducibility was then assessed using an intragroup correlation coefficient. Covariate effects were included to characterise the variation of both mean and variance model parameters.

# 1 | INTRODUCTION

The current race to publish has led to a substantial increase in scientific articles published each year and raises issues in terms of scientific integrity and value. Owing to their high level of evidence, meta-analyses have become common practice in clinical research, and are therefore not exempt from this rush to publish.¹,² In particular, the number of meta-analyses of aggregate data has dramatically increased owing to the ease of extracting data from publications and the development of free, user-friendly and dedicated statistical software. This phenomenon has resulted in the replication of meta-analyses, with several analyses having the same objectives and including the same studies. It is often described in the literature as redundancy and has already been observed in many fields of medical research.³⁻¹⁶

Although investigating the same outcomes and including the same studies, these meta-analyses may differ in terms of the population of interest or the statistical methodology used, and these differences may influence their results. This phenomenon may be regarded as a 'vibration of effects' (VoE) and describes the extent to which the results of analyses could change according to the characteristics of the population included or the methodological choices made.¹⁷

The VoE phenomenon is classically investigated in a single research project by artificially producing multiple

meta-analyses from available studies.¹⁸,¹⁹ Assessment of the replication of meta-analysis results provides an example of VoE 'naturally' observed in multiple research projects. So far, the variability of meta-analysis results has mostly been reported using graphical descriptive representations,¹⁹⁻²¹ indicating the magnitude of effects (i.e., relative risk, RR) on the x-axis as a function of its level of significance (i.e., its *p*-value) on the y-axis. To the best of our knowledge, no quantitative analysis has been proposed up to now. The quantitative analysis of these results could enable breakdown of the total observed variability and subsequently quantification of the variability generated by the replication process. It could also allow identification of parameters likely to explain this variability, such as methodological quality or the statistical analysis procedures used. These parameters might explain systematic mean differences between meta-analysis results and/or dispersion (variance) of these results. Such quantitative analysis is thus essential to analyse the distortion (or improvement) in treatment effect estimates created by the phenomenon of replication and to better characterise the robustness of these estimates.

This paper aims to provide a general methodological framework for quantitative analysis of the reproducibility of meta-analysis results. The evaluation of direct oral anticoagulants (DOACs) in the acute treatment of venous thromboembolism (VTE) was chosen as an illustrative example of this approach.
## 2 | ILLUSTRATIVE EXAMPLE: META-ANALYSES OF DOACS IN THE TREATMENT OF ACUTE VTE

To illustrate the general methodological framework, we applied our approach to the example of meta-analyses evaluating DOACS in relation to any comparator in patients treated for VTE during the acute or extended phase of anticoagulation. The methods used for identification and selection of meta-analyses, data extraction and methodological quality assessment are described in Supplementary Table 1. For each meta-analysis, all treatment effect estimates concerning efficacy (recurrent VTE) and/or safety (major bleeding) outcomes were extracted. The estimated relative risk (RR) and the precision of the results defined by the standard error (SE) were then determined. The SE of the RR on a log scale was derived from the 95% confidence interval of RR ($CI_{95\%} = [RR_{LB}, RR_{UB}]$) using the following formula:

$$SE(\log RR) = \frac{\log(RR_{UB}) - \log(RR_{LB})}{2 \times 1.96},$$

The efficacy and safety results extracted were pooled in a single database to be jointly analysed and to provide more power for covariate analysis.

For each result, the following characteristics were also extracted: the type of meta-analysis (pairwise or network), the type of statistical model used (fixed or random-effect), the type of analysis (original analysis or secondary analyses, for example, results derived from subgroup or sensitivity analyses), the type of inference (frequentist or Bayesian), and the inclusion of non-randomised studies (yes or no).

The replication groups for meta-analysis results were then defined according to the types of population (treatment indication) and intervention compared. The results belonging to the same replication group therefore concerned the same studies, the same outcomes, the same treatment indications, and the same treatment comparisons. Each replication group therefore contained both the original results and those of all the subsequent meta-analyses that replicated it.

## 3 | GENERAL METHODOLOGICAL FRAMEWORK

The general methodological framework for evaluating the reproducibility of meta-analyses was composed of several parts: vibration of effects, variability breakdown, determinants of variability, reproducibility quantification

and vibration of effects simulation. Each of these parts is described below.

### 3.1 | Visualisation of the vibration of effects between and within replication groups

The vibration of effects (VoE) may be graphically visualised by representing the magnitude of effects (i.e., the RR) on the x-axis as a function of its level of significance (i.e., its p-value) on the y-axis. The p-value depends on both the effect size estimate and the precision of the estimation. Within the replication framework, the vibration may be represented in two formats: (1) as raw data, with each point representing a result observed in one meta-analysis, (2) as aggregated data of individual replication groups, with each point representing the median treatment effect of a particular replication group. The variability within each replication group was visualised by representing the range (minimum and maximum) of observed values for both RR and p-value within the replication group concerned.

### 3.2 | Variability breakdown

The data used to evaluate the reproducibility of the results of meta-analyses corresponded to several statistical results (k) from the meta-analyses (j) within a particular replication group (i) (See Section 2 for the definition of a replication group). The structure of the data is illustrated in Figure 1, with I replication groups. The i-th replication group contained $J_i$ meta-analyses and the j-th meta-analysis of this group reported $K_{ij}$ results (RR and SE) and associated covariates ($c = 1,...,C$).

First, a statistical result of one meta-analysis is a bidimensional quantity that is composed of an overall estimation of the treatment effect and its corresponding uncertainty, and these may be highly interdependent. To properly characterise such data, these two quantities need to be jointly analysed within the same model to take into account their interdependency. In the framework of meta-analysis results, reported values of treatment effect (RR) and its corresponding standard error (SE, representing the uncertainty of the estimation) were used and analysed within a bivariate model.

Second, the data structure needs to be defined in order to break down the total observed variability into variability between replication groups and variability within individual replication groups. For that purpose, a nested random effect structure must be defined due to the hierarchical nature of the data. In the illustrative
## 2 | ILLUSTRATIVE EXAMPLE: META-ANALYSES OF DOACS IN THE TREATMENT OF ACUTE VTE

To illustrate the general methodological framework, we applied our approach to the example of meta-analyses evaluating DOACS in relation to any comparator in patients treated for VTE during the acute or extended phase of anticoagulation. The methods used for identification and selection of meta-analyses, data extraction and methodological quality assessment are described in Supplementary Table 1. For each meta-analysis, all treatment effect estimates concerning efficacy (recurrent VTE) and/or safety (major bleeding) outcomes were extracted. The estimated relative risk (RR) and the precision of the results defined by the standard error (SE) were then determined. The SE of the RR on a log scale was derived from the 95% confidence interval of RR ($CI_{95\%} = [RR_{LB}, RR_{UB}]$) using the following formula:

$$SE(\log RR) = \frac{\log(RR_{UB}) - \log(RR_{LB})}{2 \times 1.96},$$

The efficacy and safety results extracted were pooled in a single database to be jointly analysed and to provide more power for covariate analysis.

For each result, the following characteristics were also extracted: the type of meta-analysis (pairwise or network), the type of statistical model used (fixed or random-effect), the type of analysis (original analysis or secondary analyses, for example, results derived from subgroup or sensitivity analyses), the type of inference (frequentist or Bayesian), and the inclusion of non-randomised studies (yes or no).

The replication groups for meta-analysis results were then defined according to the types of population (treatment indication) and intervention compared. The results belonging to the same replication group therefore concerned the same studies, the same outcomes, the same treatment indications, and the same treatment comparisons. Each replication group therefore contained both the original results and those of all the subsequent meta-analyses that replicated it.

## 3 | GENERAL METHODOLOGICAL FRAMEWORK

The general methodological framework for evaluating the reproducibility of meta-analyses was composed of several parts: vibration of effects, variability breakdown, determinants of variability, reproducibility quantification

and vibration of effects simulation. Each of these parts is described below.

### 3.1 | Visualisation of the vibration of effects between and within replication groups

The vibration of effects (VoE) may be graphically visualised by representing the magnitude of effects (i.e., the RR) on the x-axis as a function of its level of significance (i.e., its p-value) on the y-axis. The p-value depends on both the effect size estimate and the precision of the estimation. Within the replication framework, the vibration may be represented in two formats: (1) as raw data, with each point representing a result observed in one meta-analysis, (2) as aggregated data of individual replication groups, with each point representing the median treatment effect of a particular replication group. The variability within each replication group was visualised by representing the range (minimum and maximum) of observed values for both RR and p-value within the replication group concerned.

### 3.2 | Variability breakdown

The data used to evaluate the reproducibility of the results of meta-analyses corresponded to several statistical results (k) from the meta-analyses (j) within a particular replication group (i) (See Section 2 for the definition of a replication group). The structure of the data is illustrated in Figure 1, with I replication groups. The i-th replication group contained $J_i$ meta-analyses and the j-th meta-analysis of this group reported $K_{ij}$ results (RR and SE) and associated covariates ($c = 1,...,C$).

First, a statistical result of one meta-analysis is a bidimensional quantity that is composed of an overall estimation of the treatment effect and its corresponding uncertainty, and these may be highly interdependent. To properly characterise such data, these two quantities need to be jointly analysed within the same model to take into account their interdependency. In the framework of meta-analysis results, reported values of treatment effect (RR) and its corresponding standard error (SE, representing the uncertainty of the estimation) were used and analysed within a bivariate model.

Second, the data structure needs to be defined in order to break down the total observed variability into variability between replication groups and variability within individual replication groups. For that purpose, a nested random effect structure must be defined due to the hierarchical nature of the data. In the illustrative
<figure><img src="image_3.png" /><figcaption>FIGURE 1 Schematic representation of the data structure of meta-analysis results representing i (i = 1, ..., I) replication groups containing j = 1, ..., Ji meta-analyses which include k = 1, ..., Kij results. [Colour figure can be viewed at wileyonlinelibrary.com]</figcaption></figure>

example, a two-level nested structure was used to evaluate the meta-analyses within replication groups.

The following bivariate model with the nested random effect on the intercept was used:

with i, j and k denoting the k-th results of the j-th meta-analysis within the i-th replication group, respectively.

This model enables breakdown of the total variability into three components: (i) the variability between replica-

$$ 
\begin{gathered}
\left( \begin{array}{c} u_i^{RR} \\ u_i^{SE} \end{array} \right) \sim \mathcal{N} \left( 0, \begin{bmatrix} \omega_{u,RR}^2 & \rho_u \omega_{u,RR} \omega_{u,SE} \\ \rho_u \omega_{u,RR} \omega_{u,SE} & \omega_{u,SE}^2 \end{bmatrix} \right), \left( \begin{array}{c} v_{j|i}^{RR} \\ v_{j|i}^{SE} \end{array} \right) \sim \mathcal{N} \left( 0, \begin{bmatrix} \omega_{v,RR}^2 & \rho_v \omega_{v,RR} \omega_{v,SE} \\ \rho_v \omega_{v,RR} \omega_{v,SE} & \omega_{v,SE}^2 \end{bmatrix} \right), \\
\left( \begin{array}{c} \varepsilon_{ijk}^{RR} \\ \varepsilon_{ijk}^{SE} \end{array} \right) \sim \mathcal{N} \left( 0, \begin{bmatrix} \sigma_{\varepsilon,RR}^2 & \rho_{\varepsilon} \sigma_{\varepsilon,RR} \sigma_{\varepsilon,SE} \\ \rho_{\varepsilon} \sigma_{\varepsilon,RR} \sigma_{\varepsilon,SE} & \sigma_{\varepsilon,SE}^2 \end{bmatrix} \right)
\end{gathered}
$$

which may be summarised as follows:

$$ \left( \begin{array}{c} \log RR_{ijk} \\ \log SE_{ijk} \end{array} \right) \sim N \left( \left( \begin{array}{c} \beta_0^{RR} \\ \beta_0^{SE} \end{array} \right), \Sigma \right), $$

with

$$ \sum = \left( \begin{array}{c} \omega_{u,RR}^2 + \omega_{v,RR}^2 + \sigma_{\varepsilon,RR}^2 \\ \rho_u \omega_{u,RR} \omega_{u,SE} + \rho_v \omega_{v,RR} \omega_{v,SE} + \rho_{\varepsilon} \sigma_{\varepsilon,RR} \sigma_{\varepsilon,SE} \end{array} \right) \left( \begin{array}{c} \rho_u \omega_{u,RR} \omega_{u,SE} + \rho_v \omega_{v,RR} \omega_{v,SE} + \rho_{\varepsilon} \sigma_{\varepsilon,RR} \sigma_{\varepsilon,SE} \\ \omega_{u,SE}^2 + \omega_{v,SE}^2 + \sigma_{\varepsilon,SE}^2 \end{array} \right), $$

tion groups ($u_i$), (ii) the variability within each replication group ($v_{j|i}$) and (iii) the variability within each meta-analysis (residual variability, $\varepsilon_{ijk}$).

For each numerical outcome (RR or SE) of a meta-analysis, the parameter $\beta_0$ corresponds to the intercept. The parameters $u_i$ and $v_{j|i}$ represent the random effect of the i-th replication group and the random effect of the

[Correction added on 07 December 2023, after first online publication: The above equation has been corrected in this version.]
<figure><img src="image_3.png" /><figcaption>FIGURE 1 Schematic representation of the data structure of meta-analysis results representing i (i = 1, ..., I) replication groups containing j = 1, ..., Ji meta-analyses which include k = 1, ..., Kij results. [Colour figure can be viewed at wileyonlinelibrary.com]</figcaption></figure>

example, a two-level nested structure was used to evaluate the meta-analyses within replication groups.

The following bivariate model with the nested random effect on the intercept was used:

with i, j and k denoting the k-th results of the j-th meta-analysis within the i-th replication group, respectively.

This model enables breakdown of the total variability into three components: (i) the variability between replica-

$$ 
\begin{gathered}
\left( \begin{array}{c} u_i^{RR} \\ u_i^{SE} \end{array} \right) \sim \mathcal{N} \left( 0, \begin{bmatrix} \omega_{u,RR}^2 & \rho_{u,\omega_{u,RR}\omega_{u,SE}} \\ \rho_{u,\omega_{u,RR}\omega_{u,SE}} & \omega_{u,SE}^2 \end{bmatrix} \right), \left( \begin{array}{c} v_{j|i}^{RR} \\ v_{j|i}^{SE} \end{array} \right) \sim \mathcal{N} \left( 0, \begin{bmatrix} \omega_{v,RR}^2 & \rho_{v,\omega_{v,RR}\omega_{v,SE}} \\ \rho_{v,\omega_{v,RR}\omega_{v,SE}} & \omega_{v,SE}^2 \end{bmatrix} \right), \\
\left( \begin{array}{c} \varepsilon_{ijk}^{RR} \\ \varepsilon_{ijk}^{SE} \end{array} \right) \sim \mathcal{N} \left( 0, \begin{bmatrix} \sigma_{\varepsilon,RR}^2 & \rho_{\varepsilon,\sigma_{\varepsilon,RR}\sigma_{\varepsilon,SE}} \\ \rho_{\varepsilon,\sigma_{\varepsilon,RR}\sigma_{\varepsilon,SE}} & \sigma_{\varepsilon,SE}^2 \end{bmatrix} \right)
\end{gathered}
$$

which may be summarised as follows:

$$ \left( \begin{array}{c} \log RR_{ijk} \\ \log SE_{ijk} \end{array} \right) \sim N \left( \left( \begin{array}{c} \beta_0^{RR} \\ \beta_0^{SE} \end{array} \right), \Sigma \right), $$

with

$$ \sum = \left( \begin{array}{c} \omega_{u,RR}^2 + \omega_{v,RR}^2 + \sigma_{\varepsilon,RR}^2 \\ \rho_{u,\omega_{u,RR}\omega_{u,SE}} + \rho_{v,\omega_{v,RR}\omega_{v,SE}} + \rho_{\varepsilon,\sigma_{\varepsilon,RR}\sigma_{\varepsilon,SE}} \end{array} \right) \left( \begin{array}{c} \rho_{u,\omega_{u,RR}\omega_{u,SE}} + \rho_{v,\omega_{v,RR}\omega_{v,SE}} + \rho_{\varepsilon,\sigma_{\varepsilon,RR}\sigma_{\varepsilon,SE}} \\ \omega_{u,SE}^2 + \omega_{v,SE}^2 + \sigma_{\varepsilon,SE}^2 \end{array} \right), $$

tion groups ($u_i$), (ii) the variability within each replication group ($v_{j|i}$) and (iii) the variability within each meta-analysis (residual variability, $\varepsilon_{ijk}$).

For each numerical outcome (RR or SE) of a meta-analysis, the parameter $\beta_0$ corresponds to the intercept. The parameters $u_i$ and $v_{j|i}$ represent the random effect of the i-th replication group and the random effect of the

[Correction added on 07 December 2023, after first online publication: The above equation has been corrected in this version.]
j-th meta-analysis nested in the i-th replication group, respectively. The standard deviation $\omega_u$ quantifies the variability between replication groups, the standard deviation $\omega_v$ quantifying the variability between meta-analyses within each replication group. Finally, $\epsilon_{ijk}$ is the residual error (within meta-analysis variability) for the k-th result of the j-th meta-analysis within the i-th replication group with $\sigma_e^2$ quantifying the within meta-analysis variability. The parameters $\rho_u$, $\rho_v$ and $\rho_e$ represent the correlation of the random effects and the residual error between the two types of numerical outcomes (RR and SE).

## 3.3 | Determinants of variability

To investigate the effect of meta-analysis characteristics on result reproducibility, we extended the previous model to a meta-regression that included the covariates used for modelling both mean parameters and residual variance parameters of the overall treatment effect (RR) and its uncertainty (SE):

1. Investigation of the impact of C binary covariates on mean parameter value by including these covariates as predictors. The parameters $\beta_c$ ($c=1,...,C$) quantify the systematic difference in mean meta-analysis results between levels of the c-th covariate.

$$ \left\{ \begin{aligned} \log(RR_{ijk}) &= \beta_0^{RR} + u_i^{RR} + v_{ji}^{RR} + \sum_{c=1}^{C} \beta_c^{RR} \times X_{cij} + \epsilon_{ijk}^{RR} \\ \log(SE_{ijk}) &= \beta_0^{SE} + u_i^{SE} + v_{ji}^{SE} + \sum_{c=1}^{C} \beta_c^{SE} \times X_{cij} + \epsilon_{ijk}^{SE} \end{aligned} \right. $$

2. Investigation of the impact of C binary covariates on within meta-analysis variability by stratifying residual variance on covariate levels. Residual variability was modelled using the following formula:

$$ \left( \begin{array}{c} \epsilon_{ijk}^{RR} \\ \epsilon_{ijk}^{SE} \end{array} \right) \sim N \left( 0, \begin{bmatrix} \sigma_{\epsilon,RR}^2(X_{ij}) & \rho_e \sigma_{\epsilon,RR}(X_{ij}) \sigma_{\epsilon,SE}(X_{ij}) \\ \rho_e \sigma_{\epsilon,RR}(X_{ij}) \sigma_{\epsilon,SE}(X_{ij}) & \sigma_{\epsilon,SE}^2(X_{ij}) \end{bmatrix} \right), \\ \sigma_{\epsilon,RR}(X) = \sigma_{\epsilon,RR} \times e^{\sum_{c=1}^{C} \gamma_c^{RR} \times X_{cij}}, \text{ and } \sigma_{\epsilon,SE}(X) = \sigma_{\epsilon,SE} \times e^{\sum_{c=1}^{C} \gamma_c^{SE} \times X_{cij}}. $$

The parameters $\gamma_c$ quantify systematic difference in terms of variability of meta-analysis results between levels of the c-th covariates.

The impact of covariates was expressed as the ratio of change from the reference value using the exponential of each coefficient (i.e., $e^{\beta_c}$ and $e^{\gamma_c}$). The covariates assessed concerned: the nature of the results (efficacy vs. safety), the type of analysis (secondary vs. original analysis), the meta-

analysis design (network vs. pairwise), the temporality of the meta-analyses (replicated vs. original), the overlap in authorship (no [i.e., if a meta-analysis shared no common author with other meta-analyses] vs. yes), the risk of bias of the meta-analyses (high vs. low), the nature of the statistical analysis (Bayesian versus frequentist), the statistical model used to combine the results (random-effects model vs. fixed-effect model), the design of the studies included (inclusion of observational studies with or without randomised controlled trials (RCTs) vs. RCTs only). Using this model, two analyses were performed. The first analysis involved adjustment only on the nature of the result (safety vs. efficacy) and the type of analysis (secondary vs. original analysis) to quantify the reproducibility within replication groups. The second analysis involved adjustment on all the covariates described above to investigate factors that could potentially explain the differences between meta-analyses.

## 3.4 | Reproducibility quantification

The reproducibility of meta-analyses within replication groups could be quantified using the following intra-replication group correlation coefficient (IRGC):

$$ IRGC(X) = \frac{\omega_u^2}{\omega_u^2 + \omega_v^2 + \sigma_e^2(X)}. $$

The IRGC was calculated separately for RR and SE. This coefficient varies between 0 and 1. Values below 0.5 indicate poor reproducibility, between 0.5 and 0.75 moderate reproducibility, between 0.75 and 0.9 good reproducibility, any value above 0.9 indicating very good reproducibility.²² For RR, the IRGC is interpreted as the percentage of variability of the estimation of the treatment effect that is explained by the replication group, the closer the IRGC of the RR is to 1, the more reproducible the treatment effect within replication groups. For SE, the IRGC is interpreted as the percentage of variability of the precision of the estimation that is explained by the replication group, the closer the IRGC of the SE is to 1, the more reproducible the precision of the estimation within replication groups.

## 3.5 | Vibration of effect simulations

To visualise the impact of variability within replication groups and variability of the variation in overall treatment effect and its statistical significance, we conducted a simulation study for generating VoE plots from the bivariate normal mixed-effect model described previously. This generation was a two-step process.
j-th meta-analysis nested in the i-th replication group, respectively. The standard deviation $\omega_u$ quantifies the variability between replication groups, the standard deviation $\omega_v$ quantifying the variability between meta-analyses within each replication group. Finally, $\epsilon_{ijk}$ is the residual error (within meta-analysis variability) for the k-th result of the j-th meta-analysis within the i-th replication group with $\sigma_e^2$ quantifying the within meta-analysis variability. The parameters $\rho_u$, $\rho_v$ and $\rho_e$ represent the correlation of the random effects and the residual error between the two types of numerical outcomes (RR and SE).

## 3.3 | Determinants of variability

To investigate the effect of meta-analysis characteristics on result reproducibility, we extended the previous model to a meta-regression that included the covariates used for modelling both mean parameters and residual variance parameters of the overall treatment effect (RR) and its uncertainty (SE):

1. Investigation of the impact of C binary covariates on mean parameter value by including these covariates as predictors. The parameters $\beta_c$ ($c=1,...,C$) quantify the systematic difference in mean meta-analysis results between levels of the c-th covariate.

$$ 
\left\{ 
\begin{aligned}
\log(RR_{ijk}) &= \beta_0^{RR} + u_i^{RR} + v_{ji}^{RR} + \sum_{c=1}^{C} \beta_c^{RR} \times X_{cij} + \epsilon_{ijk}^{RR} \\
\log(SE_{ijk}) &= \beta_0^{SE} + u_i^{SE} + v_{ji}^{SE} + \sum_{c=1}^{C} \beta_c^{SE} \times X_{cij} + \epsilon_{ijk}^{SE}
\end{aligned}
\right. 
$$

2. Investigation of the impact of C binary covariates on within meta-analysis variability by stratifying residual variance on covariate levels. Residual variability was modelled using the following formula:

$$ 
\left( \begin{array}{c} \epsilon_{ijk}^{RR} \\ \epsilon_{ijk}^{SE} \end{array} \right) \sim \mathcal{N} \left( 0, \begin{array}{cc} \sigma_{\epsilon,RR}^2(X_{ij}) & \rho_{\epsilon} \sigma_{\epsilon,RR}(X_{ij}) \sigma_{\epsilon,SE}(X_{ij}) \\ \rho_{\epsilon} \sigma_{\epsilon,RR}(X_{ij}) \sigma_{\epsilon,SE}(X_{ij}) & \sigma_{\epsilon,SE}^2(X_{ij}) \end{array} \right), \\
\sigma_{\epsilon,RR}(X) = \sigma_{\epsilon,RR} \times e^{\sum_{c=1}^{C} \gamma_c^{RR} \times X_{cij}}, \quad \text{and} \quad \sigma_{\epsilon,SE}(X) = \sigma_{\epsilon,SE} \times e^{\sum_{c=1}^{C} \gamma_c^{SE} \times X_{cij}}.
$$

The parameters $\gamma_c$ quantify systematic difference in terms of variability of meta-analysis results between levels of the c-th covariates.

The impact of covariates was expressed as the ratio of change from the reference value using the exponential of each coefficient (i.e., $e^{\beta_c}$ and $e^{\gamma_c}$). The covariates assessed concerned: the nature of the results (efficacy vs. safety), the type of analysis (secondary vs. original analysis), the meta-

analysis design (network vs. pairwise), the temporality of the meta-analyses (replicated vs. original), the overlap in authorship (no [i.e., if a meta-analysis shared no common author with other meta-analyses] vs. yes), the risk of bias of the meta-analyses (high vs. low), the nature of the statistical analysis (Bayesian versus frequentist), the statistical model used to combine the results (random-effects model vs. fixed-effect model), the design of the studies included (inclusion of observational studies with or without randomised controlled trials (RCTs) vs. RCTs only). Using this model, two analyses were performed. The first analysis involved adjustment only on the nature of the result (safety vs. efficacy) and the type of analysis (secondary vs. original analysis) to quantify the reproducibility within replication groups. The second analysis involved adjustment on all the covariates described above to investigate factors that could potentially explain the differences between meta-analyses.

## 3.4 | Reproducibility quantification

The reproducibility of meta-analyses within replication groups could be quantified using the following intra-replication group correlation coefficient (IRGC):

$$ IRGC(X) = \frac{\omega_u^2}{\omega_u^2 + \omega_v^2 + \sigma_e^2(X)}. $$

The IRGC was calculated separately for RR and SE. This coefficient varies between 0 and 1. Values below 0.5 indicate poor reproducibility, between 0.5 and 0.75 moderate reproducibility, between 0.75 and 0.9 good reproducibility, any value above 0.9 indicating very good reproducibility.²² For RR, the IRGC is interpreted as the percentage of variability of the estimation of the treatment effect that is explained by the replication group, the closer the IRGC of the RR is to 1, the more reproducible the treatment effect within replication groups. For SE, the IRGC is interpreted as the percentage of variability of the precision of the estimation that is explained by the replication group, the closer the IRGC of the SE is to 1, the more reproducible the precision of the estimation within replication groups.

## 3.5 | Vibration of effect simulations

To visualise the impact of variability within replication groups and variability of the variation in overall treatment effect and its statistical significance, we conducted a simulation study for generating VoE plots from the bivariate normal mixed-effect model described previously. This generation was a two-step process.
The first step was to simulate RR and SE values for a dedicated replication group using the formula:

(the list of references of the meta-analyses included is provided in the Supplementary files). Among these

$$ \begin{bmatrix} \log(RR_{sim}) \\ \log(SE_{sim}) \end{bmatrix} \sim N \left( \begin{bmatrix} \beta_0^{RR} + \sum_{c=1}^{C} \beta_c^{RR} \times X_c + u^{RR} \\ \beta_0^{SE} + \sum_{c=1}^{C} \beta_c^{SE} \times X_c + u^{SE} \end{bmatrix}, \begin{bmatrix} \omega_{v,RR}^2 + \sigma_{\epsilon,RR}^2(X) & \rho_v \omega_{v,RR} \omega_{v,SE} + \rho_{\epsilon} \sigma_{\epsilon,RR} \sigma_{\epsilon,SE} \\ \rho_v \omega_{v,RR} \omega_{v,SE} + \rho_{\epsilon} \sigma_{\epsilon,RR} \sigma_{\epsilon,SE} & \omega_{v,SE}^2 + \sigma_{\epsilon,SE}^2(X) \end{bmatrix} \right) $$

where $u^{RR}$ and $u^{SE}$ are random effects specific to the replication group studied. For the simulation, they were set at a value specific to the desired replication group.

The second step was to calculate a corresponding p-value based on the simulated values, using the following formula:

$$ p_{sim} = 2 \times \left( 1 - \Phi \left( \left| \frac{\log(RR_{sim})}{SE_{sim}} \right| \right) \right), $$

where $\Phi$ is the cumulative distribution function of the standard normal distribution $N(0,1)$.

Simulated VoE could be then visualised using a heat map plotting the two-dimensional density of simulated $p_{sim}$ on the y-axis and $RR_{sim}$ values on the x-axis. The width of the area enclosed describes the variability of RR: the wider the area enclosed the more variable the treatment effect. The height of the area enclosed describes the variability of $p_{sim}$ and thus the variability of result uncertainty.

All estimates and their 95% credible intervals (95% CrI) were based on a Bayesian inference method with prior distributions and performed using the rstan package²³ of R statistical software. The prior distribution of regression coefficients was a normal distribution $N(0,4)$ and the prior distribution of variance components was half-normal. The number of chains was 2, the number of burn-in samples was 10,000 and the number of total MCMC iterations was 20,000. Diagnostic plots were used to assess convergence. The STAN code, both for univariate and multivariate model are provided in Supplementary Table 2.

# 4 | APPLICATION TO META-ANALYSES OF DOACS IN THE ACUTE TREATMENT OF VTE

## 4.1 | Description of the meta-analyses included

The search procedure identified 1258 potentially eligible meta-analyses, of which 144 were included in our review

144 meta-analyses, 119 assessed recurrent VTE and/or major bleeding outcomes, for a total of 857 results on recurrent VTE and 690 results on major bleeding. For efficacy, the results of the meta-analyses were split into 319 replication groups with a mean of 2.7 results per replication group (interquartile range from 1 to 3) and a maximum of 28 results for one replication group. For safety, the results of the meta-analyses were split into 295 replication groups with a mean of 2.4 results per replication group (interquartile range from 1 to 3) and a maximum of 23 results for one replication group. The median overlap between replication groups was 50.0% (interquartile range from 40.0% to 68.4%).

## 4.2 | Vibrations of effects of meta-analyses results

In the illustrative example of meta-analyses evaluating DOACS in the acute treatment of VTE, a median of 4 results (interquartile range from 1 to 9 both for efficacy and safety) per meta-analysis were available. VoE plots were stratified according to the nature of the outcomes (safety or efficacy), the type of comparator (DOACS or LMWH/VKA) and the type of results (original or secondary). A greater variability was observed concerning safety results compared to efficacy results (Figure 2 top panel for raw data and Figure 2 bottom panel for aggregated data) whatever the stratification factor (type of comparator or type of results).

With regard to raw data, when DOACS were compared to LMWH or VKA, the median treatment effect for recurrent VTE measured in the meta-analyses was 0.80 (interquartile range from 0.62 to 0.96 in favour of DOACS) and the median treatment effect for major bleeding was 0.83 (interquartile range from 0.55 to 1.33 in favour of LMWH and VKA) (Figure 2, top panel). When DOACS were compared to another DOAC, the median treatment effects for recurrent VTE and major bleeding were 1.03 (interquartile range from 0.86 to 1.25)
The first step was to simulate RR and SE values for a dedicated replication group using the formula:

(the list of references of the meta-analyses included is provided in the Supplementary files). Among these

$$ \begin{bmatrix} \log(RR_{sim}) \\ \log(SE_{sim}) \end{bmatrix} \sim N \left( \begin{bmatrix} \beta_0^{RR} + \sum_{c=1}^{C} \beta_c^{RR} \times X_c + u^{RR} \\ \beta_0^{SE} + \sum_{c=1}^{C} \beta_c^{SE} \times X_c + u^{SE} \end{bmatrix}, \begin{bmatrix} \omega_{v,RR}^2 + \sigma_{\varepsilon,RR}^2(X) & \rho_v \omega_{v,RR} \omega_{v,SE} + \rho_{\varepsilon} \sigma_{\varepsilon,RR} \sigma_{\varepsilon,SE} \\ \rho_v \omega_{v,RR} \omega_{v,SE} + \rho_{\varepsilon} \sigma_{\varepsilon,RR} \sigma_{\varepsilon,SE} & \omega_{v,SE}^2 + \sigma_{\varepsilon,SE}^2(X) \end{bmatrix} \right) $$

where $u^{RR}$ and $u^{SE}$ are random effects specific to the replication group studied. For the simulation, they were set at a value specific to the desired replication group.

The second step was to calculate a corresponding p-value based on the simulated values, using the following formula:

$$ p_{sim} = 2 \times \left( 1 - \Phi \left( \left| \frac{\log(RR_{sim})}{SE_{sim}} \right| \right) \right), $$

where $\Phi$ is the cumulative distribution function of the standard normal distribution $N(0,1)$.

Simulated VoE could be then visualised using a heat map plotting the two-dimensional density of simulated $p_{sim}$ on the y-axis and $RR_{sim}$ values on the x-axis. The width of the area enclosed describes the variability of RR: the wider the area enclosed the more variable the treatment effect. The height of the area enclosed describes the variability of $p_{sim}$ and thus the variability of result uncertainty.

All estimates and their 95% credible intervals (95% CrI) were based on a Bayesian inference method with prior distributions and performed using the rstan package²³ of R statistical software. The prior distribution of regression coefficients was a normal distribution $N(0,4)$ and the prior distribution of variance components was half-normal. The number of chains was 2, the number of burn-in samples was 10,000 and the number of total MCMC iterations was 20,000. Diagnostic plots were used to assess convergence. The STAN code, both for univariate and multivariate model are provided in Supplementary Table 2.

# 4 | APPLICATION TO META-ANALYSES OF DOACS IN THE ACUTE TREATMENT OF VTE

## 4.1 | Description of the meta-analyses included

The search procedure identified 1258 potentially eligible meta-analyses, of which 144 were included in our review

144 meta-analyses, 119 assessed recurrent VTE and/or major bleeding outcomes, for a total of 857 results on recurrent VTE and 690 results on major bleeding. For efficacy, the results of the meta-analyses were split into 319 replication groups with a mean of 2.7 results per replication group (interquartile range from 1 to 3) and a maximum of 28 results for one replication group. For safety, the results of the meta-analyses were split into 295 replication groups with a mean of 2.4 results per replication group (interquartile range from 1 to 3) and a maximum of 23 results for one replication group. The median overlap between replication groups was 50.0% (interquartile range from 40.0% to 68.4%).

## 4.2 | Vibrations of effects of meta-analyses results

In the illustrative example of meta-analyses evaluating DOACS in the acute treatment of VTE, a median of 4 results (interquartile range from 1 to 9 both for efficacy and safety) per meta-analysis were available. VoE plots were stratified according to the nature of the outcomes (safety or efficacy), the type of comparator (DOACS or LMWH/VKA) and the type of results (original or secondary). A greater variability was observed concerning safety results compared to efficacy results (Figure 2 top panel for raw data and Figure 2 bottom panel for aggregated data) whatever the stratification factor (type of comparator or type of results).

With regard to raw data, when DOACS were compared to LMWH or VKA, the median treatment effect for recurrent VTE measured in the meta-analyses was 0.80 (interquartile range from 0.62 to 0.96 in favour of DOACS) and the median treatment effect for major bleeding was 0.83 (interquartile range from 0.55 to 1.33 in favour of LMWH and VKA) (Figure 2, top panel). When DOACS were compared to another DOAC, the median treatment effects for recurrent VTE and major bleeding were 1.03 (interquartile range from 0.86 to 1.25)
<figure><img src="image_4.png" /><figcaption>FIGURE 2 Vibration of efficacy results (recurrent venous thromboembolism) on the left and safety results (major bleeding) on the right according to the comparator. In the upper panels, each point represents a result observed in one meta-analysis. In the lower panels, each point represents aggregated data within replication groups, that is, the median of treatment effect within replication groups. The horizontal line represents the range (minimum and maximum) of the treatment effect observed within replication groups, the vertical line representing the range of the p-values within replication groups (on a log scale for the y-axis). The dashed vertical line represents the null effect of the relative risk (RR = 1), the dashed horizontal line representing the cut-off of statistical significance (*p*-value = 0.05). Original: original results in meta-analyses. Secondary: subgroup or sensitivity results in meta-analyses. DOAC: direct oral anticoagulant; LMWH: low-molecular-weight heparin; VKA: vitamin-K antagonist. [Colour figure can be viewed at wileyonlinelibrary.com]</figcaption></figure>

and 0.40 (interquartile range from 0.14 to 0.89), respectively (Figure 2, top panel).

Concerning aggregate data, within the same replication group, when DOACs were compared to LMWH or VKA, the median average treatment effect for recurrent VTE within replication groups was 0.82 (interquartile

range from 0.66 to 0.97) and the median average treatment effect for major bleeding within replication groups was 0.72 (interquartile range from 0.54 to 1.23 (Figure 2, bottom panel). When DOACs were compared to another DOAC, the median average treatment effect within replication groups was 1.02 (interquartile range from 0.93 to
<figure><img src="image_4.png" /><figcaption>FIGURE 2 Vibration of efficacy results (recurrent venous thromboembolism) on the left and safety results (major bleeding) on the right according to the comparator. In the upper panels, each point represents a result observed in one meta-analysis. In the lower panels, each point represents aggregated data within replication groups, that is, the median of treatment effect within replication groups. The horizontal line represents the range (minimum and maximum) of the treatment effect observed within replication groups, the vertical line representing the range of the p-values within replication groups (on a log scale for the y-axis). The dashed vertical line represents the null effect of the relative risk (RR = 1), the dashed horizontal line representing the cut-off of statistical significance (*p*-value = 0.05). Original: original results in meta-analyses. Secondary: subgroup or sensitivity results in meta-analyses. DOAC: direct oral anticoagulant; LMWH: low-molecular-weight heparin; VKA: vitamin-K antagonist. [Colour figure can be viewed at wileyonlinelibrary.com]</figcaption></figure>

and 0.40 (interquartile range from 0.14 to 0.89), respectively (Figure 2, top panel).

Concerning aggregate data, within the same replication group, when DOACs were compared to LMWH or VKA, the median average treatment effect for recurrent VTE within replication groups was 0.82 (interquartile

range from 0.66 to 0.97) and the median average treatment effect for major bleeding within replication groups was 0.72 (interquartile range from 0.54 to 1.23 (Figure 2, bottom panel). When DOACs were compared to another DOAC, the median average treatment effect within replication groups was 1.02 (interquartile range from 0.93 to
<table><caption>TABLE 1 Analysis of the reproducibility of meta-analyses regarding efficacy (recurrent venous thromboembolism) and safety (major bleeding) results within replication groups.</caption><thead><tr><th rowspan="2">Treatment effect</th><th colspan="2">Recurrent VTE</th><th colspan="2">Major bleeding</th></tr><tr><th>Original analysis</th><th>Secondary analysis</th><th>Original analysis</th><th>Secondary analysis</th></tr></thead><tbody><tr><td>Variability—SD (95% CrI)</td><td></td><td></td><td></td><td></td></tr><tr><td>Between replication groups (ω<sub>u,RR</sub>)</td><td></td><td></td><td>0.79 (0.74–0.83)</td><td></td></tr><tr><td>Within replication groups (ω<sub>v,RR</sub>)</td><td></td><td></td><td>0.08 (0.05–0.11)</td><td></td></tr><tr><td>Within meta-analysis (σ<sub>ε,RR</sub>)</td><td>0.14 (0.12–0.15)</td><td>0.37 (0.35–0.40)</td><td>0.24 (0.22–0.26)</td><td>0.64 (0.59–0.70)</td></tr><tr><td>IRGC (95% CrI)</td><td>0.96 (0.95–0.97)</td><td>0.81 (0.78–0.83)</td><td>0.91 (0.89–0.93)</td><td>0.60 (0.54–0.65)</td></tr></tbody><thead><tr><th rowspan="2">Standard error</th><th colspan="2">Recurrent VTE</th><th colspan="2">Major bleeding</th></tr><tr><th>Original analysis</th><th>Secondary analysis</th><th>Original analysis</th><th>Secondary analysis</th></tr></thead><tbody><tr><td>Variability—SD (95% CrI)</td><td></td><td></td><td></td><td></td></tr><tr><td>Between replication groups (ω<sub>u,SE</sub>)</td><td></td><td></td><td>0.69 (0.64–0.75)</td><td></td></tr><tr><td>Within replication groups (ω<sub>v,SE</sub>)</td><td></td><td></td><td>0.27 (0.22–0.32)</td><td></td></tr><tr><td>Within meta-analysis (σ<sub>ε,SE</sub>)</td><td>0.30 (0.27–0.33)</td><td>0.44 (0.41–0.47)</td><td>0.28 (0.26–0.32)</td><td>0.42 (0.38–0.46)</td></tr><tr><td>IRGC (95% CrI)</td><td>0.75 (0.69–0.80)</td><td>0.64 (0.60–0.69)</td><td>0.76 (0.69–0.81)</td><td>0.66 (0.60–0.71)</td></tr></tbody></table>

Abbreviations: 95% CrI, 95% credible interval; IRGC, intra-replication group correlation coefficient; SD, standard deviation; VTE, venous thromboembolism.

1.23) for recurrent VTE and 0.39 (interquartile range from 0.11 to 0.89) for major bleeding (Figure 2, bottom panel).

## 4.3 | Breakdown of the variability

The proposed model (see Sections 3.2 and 3.3) was applied to meta-analyses evaluating DOACs with inclusion of the nature of the result (efficacy vs. safety) and the type of analysis (secondary vs. original analyses) as covariates to stratify the analysis and assess reproducibility with respect to these characteristics.

Regarding the treatment effect, the standard deviation for variability between replication groups ($\omega_{u,RR}$) and within replication groups ($\omega_{v,RR}$) was 0.79 (95% CrI, 0.74–0.83) and 0.08 (0.05–0.11), respectively (Table 1). For the efficacy outcome, the variability within meta-analyses was 0.14 (0.12–0.15) and 0.37 (0.35–0.40) in original and secondary analyses, respectively. This corresponded to an IRGC of 0.96 (0.95–0.97) and of 0.81 (0.78–0.83), respectively. For the safety outcome, the variability within meta-analyses was 0.24 (0.22–0.26) and 0.64 (0.59–0.70) in original and secondary analyses, respectively, corresponding to an IRGC of 0.91 (0.89–0.93) and of 0.60 (0.54–0.65), respectively. So very good reproducibility of the meta-analyses within replication groups was observed for the results of meta-analyses when considering the overall treatment effect estimated on efficacy for original analyses, good reproducibility for secondary analyses of efficacy and original analyses of safety, and moderate reproducibility for secondary analyses of safety.

Regarding reported standard errors of the treatment effect, the variability of the standard deviation between replication groups ($\omega_{u,SE}$) and within replication groups ($\omega_{v,SE}$) was 0.69 (95% CrI, 0.64–0.75) and 0.27 (0.22–0.32), respectively. For the efficacy outcome, the variability within meta-analyses was 0.30 (0.27–0.33) in original analyses and 0.44 (0.41–0.47) in secondary analyses. These values corresponded to an IRGC of 0.75 (0.69–0.80) and 0.64 (0.60–0.69), respectively. For the safety outcome, the variability within meta-analyses was 0.28 (0.26–0.32) for original analyses and 0.42 (0.38–0.46) for secondary analyses corresponding to an IRGC of 0.76 (0.69–0.81) and of 0.66 (0.60–0.71), respectively. These IRGC values indicate moderate reproducibility of the results of meta-analyses when considering the reported standard errors of the overall treatment effect estimation.

## 4.4 | Investigation of sources of variability of efficacy and safety results within replication groups

None of the covariates tested had an impact on the mean treatment effect ($\beta_c^{RR}$) except for safety results compared to efficacy results (ratio 1.20, 95% CrI [1.06–1.36]), Figure 3, top left panel). In terms of treatment effect uncertainty (Figure 3, top right panel), the safety results also had an impact on the treatment effect uncertainty ($\beta_c^{SE}$) compared to efficacy results (1.62 [1.45–1.80]), as well as on the results of secondary analyses compared to original analyses (1.14 [1.09–1.20]). The treatment effect uncertainty increased depending on the meta-analytical
<table><caption>TABLE 1 Analysis of the reproducibility of meta-analyses regarding efficacy (recurrent venous thromboembolism) and safety (major bleeding) results within replication groups.</caption><thead><tr><th rowspan="2">Treatment effect</th><th colspan="2">Recurrent VTE</th><th colspan="2">Major bleeding</th></tr><tr><th>Original analysis</th><th>Secondary analysis</th><th>Original analysis</th><th>Secondary analysis</th></tr></thead><tbody><tr><td>Variability—SD (95% CrI)</td><td></td><td></td><td></td><td></td></tr><tr><td>Between replication groups (ω<sub>u,RR</sub>)</td><td></td><td></td><td>0.79 (0.74–0.83)</td><td></td></tr><tr><td>Within replication groups (ω<sub>v,RR</sub>)</td><td></td><td></td><td>0.08 (0.05–0.11)</td><td></td></tr><tr><td>Within meta-analysis (σ<sub>ε,RR</sub>)</td><td>0.14 (0.12–0.15)</td><td>0.37 (0.35–0.40)</td><td>0.24 (0.22–0.26)</td><td>0.64 (0.59–0.70)</td></tr><tr><td>IRGC (95% CrI)</td><td>0.96 (0.95–0.97)</td><td>0.81 (0.78–0.83)</td><td>0.91 (0.89–0.93)</td><td>0.60 (0.54–0.65)</td></tr></tbody></table><table><thead><tr><th rowspan="2">Standard error</th><th colspan="2">Recurrent VTE</th><th colspan="2">Major bleeding</th></tr><tr><th>Original analysis</th><th>Secondary analysis</th><th>Original analysis</th><th>Secondary analysis</th></tr></thead><tbody><tr><td>Variability—SD (95% CrI)</td><td></td><td></td><td></td><td></td></tr><tr><td>Between replication groups (ω<sub>u,SE</sub>)</td><td></td><td></td><td>0.69 (0.64–0.75)</td><td></td></tr><tr><td>Within replication groups (ω<sub>v,SE</sub>)</td><td></td><td></td><td>0.27 (0.22–0.32)</td><td></td></tr><tr><td>Within meta-analysis (σ<sub>ε,SE</sub>)</td><td>0.30 (0.27–0.33)</td><td>0.44 (0.41–0.47)</td><td>0.28 (0.26–0.32)</td><td>0.42 (0.38–0.46)</td></tr><tr><td>IRGC (95% CrI)</td><td>0.75 (0.69–0.80)</td><td>0.64 (0.60–0.69)</td><td>0.76 (0.69–0.81)</td><td>0.66 (0.60–0.71)</td></tr></tbody></table>

Abbreviations: 95% CrI, 95% credible interval; IRGC, intra-replication group correlation coefficient; SD, standard deviation; VTE, venous thromboembolism.

1.23) for recurrent VTE and 0.39 (interquartile range from 0.11 to 0.89) for major bleeding (Figure 2, bottom panel).

## 4.3 | Breakdown of the variability

The proposed model (see Sections 3.2 and 3.3) was applied to meta-analyses evaluating DOACs with inclusion of the nature of the result (efficacy vs. safety) and the type of analysis (secondary vs. original analyses) as covariates to stratify the analysis and assess reproducibility with respect to these characteristics.

Regarding the treatment effect, the standard deviation for variability between replication groups ($\omega_{u,RR}$) and within replication groups ($\omega_{v,RR}$) was 0.79 (95% CrI, 0.74–0.83) and 0.08 (0.05–0.11), respectively (Table 1). For the efficacy outcome, the variability within meta-analyses was 0.14 (0.12–0.15) and 0.37 (0.35–0.40) in original and secondary analyses, respectively. This corresponded to an IRGC of 0.96 (0.95–0.97) and of 0.81 (0.78–0.83), respectively. For the safety outcome, the variability within meta-analyses was 0.24 (0.22–0.26) and 0.64 (0.59–0.70) in original and secondary analyses, respectively, corresponding to an IRGC of 0.91 (0.89–0.93) and of 0.60 (0.54–0.65), respectively. So very good reproducibility of the meta-analyses within replication groups was observed for the results of meta-analyses when considering the overall treatment effect estimated on efficacy for original analyses, good reproducibility for secondary analyses of efficacy and original analyses of safety, and moderate reproducibility for secondary analyses of safety.

Regarding reported standard errors of the treatment effect, the variability of the standard deviation between replication groups ($\omega_{u,SE}$) and within replication groups ($\omega_{v,SE}$) was 0.69 (95% CrI, 0.64–0.75) and 0.27 (0.22–0.32), respectively. For the efficacy outcome, the variability within meta-analyses was 0.30 (0.27–0.33) in original analyses and 0.44 (0.41–0.47) in secondary analyses. These values corresponded to an IRGC of 0.75 (0.69–0.80) and 0.64 (0.60–0.69), respectively. For the safety outcome, the variability within meta-analyses was 0.28 (0.26–0.32) for original analyses and 0.42 (0.38–0.46) for secondary analyses corresponding to an IRGC of 0.76 (0.69–0.81) and of 0.66 (0.60–0.71), respectively. These IRGC values indicate moderate reproducibility of the results of meta-analyses when considering the reported standard errors of the overall treatment effect estimation.

## 4.4 | Investigation of sources of variability of efficacy and safety results within replication groups

None of the covariates tested had an impact on the mean treatment effect ($\beta_c^{RR}$) except for safety results compared to efficacy results (ratio 1.20, 95% CrI [1.06–1.36]), Figure 3, top left panel). In terms of treatment effect uncertainty (Figure 3, top right panel), the safety results also had an impact on the treatment effect uncertainty ($\beta_c^{SE}$) compared to efficacy results (1.62 [1.45–1.80]), as well as on the results of secondary analyses compared to original analyses (1.14 [1.09–1.20]). The treatment effect uncertainty increased depending on the meta-analytical
<figure><img src="image_5.png" /><figcaption>FIGURE 3 Reproducibility of efficacy (recurrent venous thromboembolism) and safety (major bleeding) results within replication groups with respect to the average variation of the treatment effect (top left) and its uncertainty (top right) and to the breakdown of the within-meta-analysis variability (residual variability) of the treatment effect (bottom left) and its uncertainty (bottom right). 95% CrI, 95% credible interval; FEM, fixed-effect model; MA, meta-analysis; NMA, network meta-analysis; REM: random-effects model; RCTs, randomised controlled trials.</figcaption></figure>

methods used: by 77% in network meta-analyses compared to pairwise meta-analyses (1.77 [1.60–2.00]), by 29% in meta-analyses using a random-effects model to combine the results compared to a fixed-effect model (1.29 [1.20–1.38]), and by 26% in meta-analyses using a Bayesian approach compared to a frequentist approach (1.26 [1.16–1.37]). A 17% increase of uncertainty was observed in meta-analyses with no overlap in authorship compared to meta-analyses with overlap in authorship (1.17 [1.07–1.26]). Conversely, treatment effect uncertainty decreased by 27% in meta-analyses that included observational studies compared to meta-analyses that included only RCTs (0.73 [0.59–0.89]).

The impact of the covariates on the within meta-analysis variance (residual variability) is illustrated in Figure 3 (bottom panel). With respect to the treatment effect ($\gamma_c^{RR}$), the residual variability was increased by 158% in secondary analyses (2.58 [2.30–2.90]), and by 88% in safety results (1.88 [1.68–2.11]). [Correction added on 07 December 2023, after first online publication: The preceding sentence has been corrected in this version.] As regards the meta-analytical methods used, the residual variability was increased by 57% in network meta-analyses (1.57 [1.36–1.80]), and by 39% in meta-analyses using a Bayesian approach (1.39 [1.20–1.61]). A 44% and 20% increase in residual variability was observed in meta-

analyses with no overlap in authorship (1.44 [1.25–1.67]) and in meta-analyses with a high risk of bias (1.20 [1.05–1.36]). Conversely, residual variability decreased by 24% in meta-analyses that included observational studies compared to meta-analyses that included only RCTs (0.76 [0.60–0.96]). With respect to the standard error of the treatment effect ($\gamma_c^{SE}$), replicated meta-analyses, meta-analyses with no overlap in authorship, meta-analyses using a random-effects model to combine the results, meta-analyses that included observational studies, and secondary analyses were associated with an increase of the residual variability, whereas network meta-analyses were associated with a decrease in residual variability (0.85 [0.75–0.97]).

## 4.5 | Vibration of effect simulations

For the random effects $u^{RR}$ and $u^{SE}$, posterior samples of each parameter shown to induce a treatment effect in a replication group were chosen from the model detailed in Section 4.4. The RR and SE values were simulated from the multivariate normal distribution described in Section 3.5. Covariates were set at specific values to represent classical scenarios of analysis. Two scenarios were simulated
<figure><img src="image_5.png" /><figcaption>FIGURE 3 Reproducibility of efficacy (recurrent venous thromboembolism) and safety (major bleeding) results within replication groups with respect to the average variation of the treatment effect (top left) and its uncertainty (top right) and to the breakdown of the within-meta-analysis variability (residual variability) of the treatment effect (bottom left) and its uncertainty (bottom right). 95% CrI, 95% credible interval; FEM, fixed-effect model; MA, meta-analysis; NMA, network meta-analysis; REM: random-effects model; RCTs, randomised controlled trials.</figcaption></figure>

methods used: by 77% in network meta-analyses compared to pairwise meta-analyses (1.77 [1.60–2.00]), by 29% in meta-analyses using a random-effects model to combine the results compared to a fixed-effect model (1.29 [1.20–1.38]), and by 26% in meta-analyses using a Bayesian approach compared to a frequentist approach (1.26 [1.16–1.37]). A 17% increase of uncertainty was observed in meta-analyses with no overlap in authorship compared to meta-analyses with overlap in authorship (1.17 [1.07–1.26]). Conversely, treatment effect uncertainty decreased by 27% in meta-analyses that included observational studies compared to meta-analyses that included only RCTs (0.73 [0.59–0.89]).

The impact of the covariates on the within meta-analysis variance (residual variability) is illustrated in Figure 3 (bottom panel). With respect to the treatment effect ($\gamma_c^{RR}$), the residual variability was increased by 158% in secondary analyses (2.58 [2.30–2.90]), and by 88% in safety results (1.88 [1.68–2.11]). [Correction added on 07 December 2023, after first online publication: The preceding sentence has been corrected in this version.] As regards the meta-analytical methods used, the residual variability was increased by 57% in network meta-analyses (1.57 [1.36–1.80]), and by 39% in meta-analyses using a Bayesian approach (1.39 [1.20–1.61]). A 44% and 20% increase in residual variability was observed in meta-

analyses with no overlap in authorship (1.44 [1.25–1.67]) and in meta-analyses with a high risk of bias (1.20 [1.05–1.36]). Conversely, residual variability decreased by 24% in meta-analyses that included observational studies compared to meta-analyses that included only RCTs (0.76 [0.60–0.96]). With respect to the standard error of the treatment effect ($\gamma_c^{SE}$), replicated meta-analyses, meta-analyses with no overlap in authorship, meta-analyses using a random-effects model to combine the results, meta-analyses that included observational studies, and secondary analyses were associated with an increase of the residual variability, whereas network meta-analyses were associated with a decrease in residual variability (0.85 [0.75–0.97]).

## 4.5 | Vibration of effect simulations

For the random effects $u^{RR}$ and $u^{SE}$, posterior samples of each parameter shown to induce a treatment effect in a replication group were chosen from the model detailed in Section 4.4. The RR and SE values were simulated from the multivariate normal distribution described in Section 3.5. Covariates were set at specific values to represent classical scenarios of analysis. Two scenarios were simulated
<figure><img src="image_6.png" /></figure>

FIGURE 4 Predictive distribution of the vibration of effects using Montecarlo simulations. Red dashed vertical lines represent the 10th and 90th percentiles of the treatment effect distribution. The continuous vertical line represents the null effect of the relative risk (RR = 1), the solid horizontal black line representing the cut-off of statistical significance (p = 0.05). [Colour figure can be viewed at wileyonlinelibrary.com]

1. A pairwise meta-analysis with a fixed-effect model using a frequentist approach in an original meta-analysis with a low risk of bias

2. A network meta-analysis with a random-effects model using a Bayesian approach in an original meta-analysis with a low risk of bias.

For each scenario, 1000 values of RR and SE were sampled. The vibration of effect generated from the simulations is shown in Figure 4. This illustrates the variability in treatment effect estimates and its impact on statistical significance within a replication group in both simulation scenarios.

The effect size was in favour of DOACs both at the 10th percentile and the 90th percentile. In network meta-analyses using a Bayesian approach and a random-effects model, the 10th percentile was 0.45 and the 90th percentile was 0.71. Regarding pairwise meta-analyses using a frequentist approach and a fixed-effect model, the 10th percentile was 0.57 and the 90th percentile was 0.73. A greater variability in treatment effect estimates and a greater uncertainty of the estimation were observed in network meta-analyses using a Bayesian approach and a random-effects model compared to frequentist, pairwise meta-analyses. This resulted in getting more conservative results in network meta-analyses.

5 | DISCUSSION

This study is the first to propose a statistical framework for analysing the replication and reproducibility of meta-analyses. Meta-analysis results defined by the overall

effect size estimation (RR) and its uncertainty (SE), were jointly analysed using a bivariate, linear, mixed-effect model to investigate the inter-dependency between RR and SE values. Covariates were included into the model at two levels: (i) the mean effect and (ii) the residual variance (within meta-analysis variability). Analysis of RR values allowed detecting covariates associated with systematic differences among meta-analyses or accounting for a bias between meta-analyses differing in methodological quality. Analysis of SE values enabled identification of covariates influencing results dispersion. It was then possible to characterise the reproducibility by quantifying the intra-replication group correlation as a function of the methodological characteristics of the meta-analyses.

The proposed statistical model was applied to meta-analyses evaluating DOACs in the acute treatment of VTE. Reproducibility of the efficacy results within replication groups was very good in original analyses and good in secondary analyses, while reproducibility of safety results within replication groups was good in original analyses and moderate in secondary analyses. Within replication groups, both the mean treatment effect and the residual variability were increased in safety results compared to efficacy results. This could be explained by the lower number of events concerning safety outcomes compared to efficacy outcomes. We also found that the treatment effect uncertainty and the residual variability of the treatment effect were increased according to the meta-analytical methods used, particularly in network meta-analyses, in meta-analyses using a Bayesian approach, and in those using a random-effects model to combine the results. These findings were expected.
<figure><img src="image_6.png" /></figure>

**FIGURE 4** Predictive distribution of the vibration of effects using Montecarlo simulations. Red dashed vertical lines represent the 10th and 90th percentiles of the treatment effect distribution. The continuous vertical line represents the null effect of the relative risk (RR = 1), the solid horizontal black line representing the cut-off of statistical significance (*p* = 0.05). [Colour figure can be viewed at wileyonlinelibrary.com]

1. A pairwise meta-analysis with a fixed-effect model using a frequentist approach in an original meta-analysis with a low risk of bias

2. A network meta-analysis with a random-effects model using a Bayesian approach in an original meta-analysis with a low risk of bias.

For each scenario, 1000 values of RR and SE were sampled. The vibration of effect generated from the simulations is shown in Figure 4. This illustrates the variability in treatment effect estimates and its impact on statistical significance within a replication group in both simulation scenarios.

The effect size was in favour of DOACs both at the 10th percentile and the 90th percentile. In network meta-analyses using a Bayesian approach and a random-effects model, the 10th percentile was 0.45 and the 90th percentile was 0.71. Regarding pairwise meta-analyses using a frequentist approach and a fixed-effect model, the 10th percentile was 0.57 and the 90th percentile was 0.73. A greater variability in treatment effect estimates and a greater uncertainty of the estimation were observed in network meta-analyses using a Bayesian approach and a random-effects model compared to frequentist, pairwise meta-analyses. This resulted in getting more conservative results in network meta-analyses.

5 | DISCUSSION

This study is the first to propose a statistical framework for analysing the replication and reproducibility of meta-analyses. Meta-analysis results defined by the overall

effect size estimation (RR) and its uncertainty (SE), were jointly analysed using a bivariate, linear, mixed-effect model to investigate the inter-dependency between RR and SE values. Covariates were included into the model at two levels: (i) the mean effect and (ii) the residual variance (within meta-analysis variability). Analysis of RR values allowed detecting covariates associated with systematic differences among meta-analyses or accounting for a bias between meta-analyses differing in methodological quality. Analysis of SE values enabled identification of covariates influencing results dispersion. It was then possible to characterise the reproducibility by quantifying the intra-replication group correlation as a function of the methodological characteristics of the meta-analyses.

The proposed statistical model was applied to meta-analyses evaluating DOACs in the acute treatment of VTE. Reproducibility of the efficacy results within replication groups was very good in original analyses and good in secondary analyses, while reproducibility of safety results within replication groups was good in original analyses and moderate in secondary analyses. Within replication groups, both the mean treatment effect and the residual variability were increased in safety results compared to efficacy results. This could be explained by the lower number of events concerning safety outcomes compared to efficacy outcomes. We also found that the treatment effect uncertainty and the residual variability of the treatment effect were increased according to the meta-analytical methods used, particularly in network meta-analyses, in meta-analyses using a Bayesian approach, and in those using a random-effects model to combine the results. These findings were expected.
Indeed, whereas in standard pairwise meta-analyses, the distribution of effect modifiers can only vary from one study to another for a particular comparison (resulting in heterogeneity), in network meta-analyses, which include different trials comparing different interventions, the distribution of effect modifiers can vary from one study to another for a particular comparison but also between comparisons with potential inconsistency. Moreover, in the illustrative example evaluating DOACs in the acute treatment of VTE, the network of treatments was weakly connected, resulting in increased uncertainty. The increase in uncertainty is also explained by the use of a Bayesian approach, in which case the phenomenon is certainly driven by the frequent use of uninformative prior distributions.

In our statistical framework, we used a Bayesian inference method with prior distributions owing to the flexibility of the Bayesian approach. The disadvantage of this approach is the long computation time, from 30 min to more than 10 h, depending on the number of iterations, the number of chains and the number of cores if the computation is parallelised. The computation time would be faster using a frequentist approach, but to date there is no software solution available for estimating such models in frequentist mode. In the model proposed, dispersion of meta-analysis results was explained by introducing covariates in analysis of the standard deviation of the residual error (intra-meta-analysis variability). For meta-analysis specific covariates, another possibility would have been to include these in analyses of the standard deviation of the between meta-analysis random effect using the following formula:

$$ \omega_{v,RR}(X) = \omega_{v,RR} \times e^{\sum_{c=1}^{c} \gamma_c^{RR} \times X_{cij}} \quad \text{and} \\ \omega_{v,SE}(X) = \omega_{v,SE} \times e^{\sum_{c=1}^{c} \gamma_c^{SE} \times X_{cij}}. $$

This would have allowed explanation of the variation of dispersion at the inter-meta-analysis level. In the example of meta-analyses evaluating DOACs, such covariate effects could not be accurately estimated due to the identifiability issue. This was certainly the consequence of a lack of between-meta-analyses data within individual replication groups. The accurate description of between-meta-analyses predictors of variability would necessitate a larger amount of data that be achieved by analysing the replication of meta-analyses for more than one drug and indication. Identifiability issues of between-group and within-group variances may also arise for datasets with too few replication groups or too few studies per

replication group. The estimation process would therefore give more weight to prior information. This situation might limit the number of covariates that could be included in the model. In this context, we advise introducing covariates into the model progressively in a step-wise manner in order to detect any over parameterization or convergence issues.

As pointed out by the reviewers, replication groups inherently have a certain overlap among the studies included in the meta-analyses. This overlap may induce a correlation between the random effects of two different replication groups. One of the limitations of the proposed approach is therefore that it cannot quantify the impact of this overlap on the correlation between the random effects of two different replication groups. One solution could be to include a correlation depending on the extent of overlap between two replication groups i and i'. This could be achieved using the following formula:

$$ \operatorname{cor}(u_i, u_{i'}) = \tanh(\rho_0 + \rho_1 \operatorname{Overlap}_{ii'}). $$

The phenomenon of meta-analysis replication, often described as redundancy in the literature, has been already highlighted in various domains.⁵,⁶,⁸,¹⁶ Several arguments have been invoked to justify redundancy, such as: the wish to update the previous meta-analysis to include data from new high-quality trials, reluctance to extrapolate the results obtained to all patients in the absence of subgroup analyses, the need to assess an endpoint not evaluated in other meta-analyses or a change in endpoint definition or its assessment time. However, the term 'redundant' implicitly signifies that replication is unnecessary from a scientific point of view. However, this interpretation is somewhat limited because the phenomenon of meta-analysis replication could also be necessary, notably to improve imperfect meta-analyses in terms of conduct, methodology or results and thus methodological quality. In our illustrative example regarding meta-analyses evaluating DOACs in the acute treatment of VTE, the treatment effect uncertainty decreased by 12% in replicated meta-analyses compared to non-replicated meta-analyses, representing an improvement in the accuracy of the results.

Nevertheless, in order to answer this question about the usefulness of replication, we need to have a broader view of this phenomenon. Ideally, analyses should be applied more broadly and more generally to several domains or drugs simultaneously (for example, immunotherapy, vaccines, etc.). However, the covariates of interest are often time-consuming to extract or evaluate. For example, the methodological quality of the meta-analysis is an important covariate but the evaluation of the level
Indeed, whereas in standard pairwise meta-analyses, the distribution of effect modifiers can only vary from one study to another for a particular comparison (resulting in heterogeneity), in network meta-analyses, which include different trials comparing different interventions, the distribution of effect modifiers can vary from one study to another for a particular comparison but also between comparisons with potential inconsistency. Moreover, in the illustrative example evaluating DOACs in the acute treatment of VTE, the network of treatments was weakly connected, resulting in increased uncertainty. The increase in uncertainty is also explained by the use of a Bayesian approach, in which case the phenomenon is certainly driven by the frequent use of uninformative prior distributions.

In our statistical framework, we used a Bayesian inference method with prior distributions owing to the flexibility of the Bayesian approach. The disadvantage of this approach is the long computation time, from 30 min to more than 10 h, depending on the number of iterations, the number of chains and the number of cores if the computation is parallelised. The computation time would be faster using a frequentist approach, but to date there is no software solution available for estimating such models in frequentist mode. In the model proposed, dispersion of meta-analysis results was explained by introducing covariates in analysis of the standard deviation of the residual error (intra-meta-analysis variability). For meta-analysis specific covariates, another possibility would have been to include these in analyses of the standard deviation of the between meta-analysis random effect using the following formula:

$$ 
\begin{align*}
\omega_{v,RR}(X) &= \omega_{v,RR} \times e^{\sum_{c=1}^{C} \gamma_c^{RR} \times X_{cij}} \quad \text{and} \\
\omega_{v,SE}(X) &= \omega_{v,SE} \times e^{\sum_{c=1}^{C} \gamma_c^{SE} \times X_{cij}}.
\end{align*}
$$

This would have allowed explanation of the variation of dispersion at the inter-meta-analysis level. In the example of meta-analyses evaluating DOACs, such covariate effects could not be accurately estimated due to the identifiability issue. This was certainly the consequence of a lack of between-meta-analyses data within individual replication groups. The accurate description of between-meta-analyses predictors of variability would necessitate a larger amount of data that be achieved by analysing the replication of meta-analyses for more than one drug and indication. Identifiability issues of between-group and within-group variances may also arise for datasets with too few replication groups or too few studies per

replication group. The estimation process would therefore give more weight to prior information. This situation might limit the number of covariates that could be included in the model. In this context, we advise introducing covariates into the model progressively in a step-wise manner in order to detect any over parameterization or convergence issues.

As pointed out by the reviewers, replication groups inherently have a certain overlap among the studies included in the meta-analyses. This overlap may induce a correlation between the random effects of two different replication groups. One of the limitations of the proposed approach is therefore that it cannot quantify the impact of this overlap on the correlation between the random effects of two different replication groups. One solution could be to include a correlation depending on the extent of overlap between two replication groups i and i'. This could be achieved using the following formula:

$$ \operatorname{cor}(u_i, u_{i'}) = \tanh(\rho_0 + \rho_1 \operatorname{Overlap}_{ii'}). $$

The phenomenon of meta-analysis replication, often described as redundancy in the literature, has been already highlighted in various domains.⁵,⁶,⁸,¹⁶ Several arguments have been invoked to justify redundancy, such as: the wish to update the previous meta-analysis to include data from new high-quality trials, reluctance to extrapolate the results obtained to all patients in the absence of subgroup analyses, the need to assess an endpoint not evaluated in other meta-analyses or a change in endpoint definition or its assessment time. However, the term 'redundant' implicitly signifies that replication is unnecessary from a scientific point of view. However, this interpretation is somewhat limited because the phenomenon of meta-analysis replication could also be necessary, notably to improve imperfect meta-analyses in terms of conduct, methodology or results and thus methodological quality. In our illustrative example regarding meta-analyses evaluating DOACs in the acute treatment of VTE, the treatment effect uncertainty decreased by 12% in replicated meta-analyses compared to non-replicated meta-analyses, representing an improvement in the accuracy of the results.

Nevertheless, in order to answer this question about the usefulness of replication, we need to have a broader view of this phenomenon. Ideally, analyses should be applied more broadly and more generally to several domains or drugs simultaneously (for example, immunotherapy, vaccines, etc.). However, the covariates of interest are often time-consuming to extract or evaluate. For example, the methodological quality of the meta-analysis is an important covariate but the evaluation of the level
of bias within a systematic review using the ROBIS tool requires substantial reflection to complete, as the assessment of each risk of bias domain requires careful analysis of the meta-analysis in order to correctly rate the risk of bias. One solution to facilitate the process of extraction and reduce the time needed for its execution would be to use artificial intelligence to perform automatic extraction of covariates of interest from the text of the publication.

## 6 | CONCLUSION

The model proposed in our study allows quantification of the reproducibility of meta-analysis results and the identification of determinants influencing the variability of the replication process. The methods used to perform the meta-analyses, and replications, in particular those with no overlap in authorship, had an impact on treatment effect and its precision that could lead to confusion in interpretation of results.

### AUTHOR CONTRIBUTIONS

**Céline Chapelle:** Conceptualization; methodology; project administration; data curation; formal analysis; software; writing – original draft; writing – review and editing. **Gwénaël Le Teuff:** Writing – review and editing. **Paul Jacques Zufferey:** Writing – review and editing. **Silvy Laporte:** Conceptualization; project administration; supervision; writing – review and editing. **Edouard Ollier:** Conceptualization; project administration; methodology; formal analysis; supervision; software; validation; writing – original draft; writing – review and editing.

### ACKNOWLEDGMENTS

This study was performed in collaboration with the F-CRIN Recap Working Group of Meta-Research. The authors thank Paula Harry for her valuable help for the English revision of this manuscript.

### FUNDING INFORMATION

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

### CONFLICT OF INTEREST STATEMENT

Céline Chapelle, Gwénaël Le Teuff, Paul Jacques Zufferey, Edouard Ollier: none. Silvy Laporte has received personal fees from Bayer for consulting, personal fees for lectures for continuing medical education from Pfizer, personal fees for lectures for continuing medical education from Eli Lilly.

### DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available from the corresponding author upon reasonable request.

### REPRODUCIBLE RESEARCH STATEMENT

Study protocol: Registered in PROSPERO (registration no. CRD42022316273) (https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=316273. Statistical code and data set: The analysis scripts are available as supporting information and data set may be requested from Céline Chapelle (e-mail: celine.chapelle@chu-st-etienne.fr).

### ORCID

Céline Chapelle https://orcid.org/0000-0003-3281-3666

Edouard Ollier https://orcid.org/0000-0001-6925-2941

### REFERENCES

1. Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. *Milbank Q.* 2016;94(3):485-514. doi:10.1111/1468-0009.12210

2. Siontis KC, Ioannidis JPA. Replication, duplication, and waste in a quarter million systematic reviews and meta-analyses. *Circ Cardiovasc Qual Outcomes.* 2018;11(12):e005212. doi:10.1161/CIROUTCOMES.118.005212

3. Riaz IB, Khan MS, Riaz H, Goldberg RJ. Disorganized systematic reviews and meta-analyses: time to systematize the conduct and publication of these study overviews? *Am J Med.* 2016;129(3):339.e11-339.e18. doi:10.1016/j.amjmed.2015.10.009

4. Bolland MJ, Grey A. A case study of discordant overlapping meta-analyses: vitamin D supplements and fracture. *PLoS ONE.* 2014;9(12):e115934. doi:10.1371/journal.pone.0115934

5. Ker K, Roberts I. Exploring redundant research into the effect of tranexamic acid on surgical bleeding: further analysis of a systematic review of randomised controlled trials. *BMJ Open.* 2015;5(8):e009460. doi:10.1136/bmjopen-2015-009460

6. Sigurdson MK, Khoury MJ, Ioannidis JPA. Redundant meta-analyses are common in genetic epidemiology. *J Clin Epidemiol.* 2020;127:40-48. doi:10.1016/j.jclinepi.2020.05.035

7. Zhang H, Tang H, He Q, et al. Surgical versus conservative intervention for acute achilles tendon rupture: a PRISMA-compliant systematic review of overlapping meta-analyses. *Medicine (Baltimore).* 2015;94(45):e1951. doi:10.1097/MD.0000000000001951

8. McDonald S, Turner S, Page MJ, Turner T. Most published systematic reviews of remdesivir for COVID-19 were redundant and lacked currency. *J Clin Epidemiol.* 2022;146:22-31. doi:10.1016/j.jclinepi.2022.02.006

9. Bolland MJ, Grey A, Reid IR. Differences in overlapping meta-analyses of vitamin D supplements and falls. *J Clin Endocrinol Metab.* 2014;99(11):4265-4272. doi:10.1210/jc.2014-2562

10. Campbell KA, Erickson BJ, Saltzman BM, et al. Is local viscosuplementation injection clinically superior to other therapies in the treatment of osteoarthritis of the knee: a systematic review of overlapping meta-analyses. *Arthrosc J Arthrosc Relat Surg.* 2015;31(10):2036-2045.e14. doi:10.1016/j.arthro.2015.03.030
of bias within a systematic review using the ROBIS tool requires substantial reflection to complete, as the assessment of each risk of bias domain requires careful analysis of the meta-analysis in order to correctly rate the risk of bias. One solution to facilitate the process of extraction and reduce the time needed for its execution would be to use artificial intelligence to perform automatic extraction of covariates of interest from the text of the publication.

## 6 | CONCLUSION

The model proposed in our study allows quantification of the reproducibility of meta-analysis results and the identification of determinants influencing the variability of the replication process. The methods used to perform the meta-analyses, and replications, in particular those with no overlap in authorship, had an impact on treatment effect and its precision that could lead to confusion in interpretation of results.

### AUTHOR CONTRIBUTIONS

**Céline Chapelle:** Conceptualization; methodology; project administration; data curation; formal analysis; software; writing – original draft; writing – review and editing. **Gwénaël Le Teuff:** Writing – review and editing. **Paul Jacques Zufferey:** Writing – review and editing. **Silvy Laporte:** Conceptualization; project administration; supervision; writing – review and editing. **Edouard Ollier:** Conceptualization; project administration; methodology; formal analysis; supervision; software; validation; writing – original draft; writing – review and editing.

### ACKNOWLEDGMENTS

This study was performed in collaboration with the F-CRIN Recap Working Group of Meta-Research. The authors thank Paula Harry for her valuable help for the English revision of this manuscript.

### FUNDING INFORMATION

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

### CONFLICT OF INTEREST STATEMENT

Céline Chapelle, Gwénaël Le Teuff, Paul Jacques Zufferey, Edouard Ollier: none. Silvy Laporte has received personal fees from Bayer for consulting, personal fees for lectures for continuing medical education from Pfizer, personal fees for lectures for continuing medical education from Eli Lilly.

### DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available from the corresponding author upon reasonable request.

### REPRODUCIBLE RESEARCH STATEMENT

Study protocol: Registered in PROSPERO (registration no. CRD42022316273) (https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=316273. Statistical code and data set: The analysis scripts are available as supporting information and data set may be requested from Céline Chapelle (e-mail: celine.chapelle@chu-st-etienne.fr).

### ORCID

Céline Chapelle https://orcid.org/0000-0003-3281-3666

Edouard Ollier https://orcid.org/0000-0001-6925-2941

### REFERENCES

1. Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. *Milbank Q.* 2016;94(3):485-514. doi:10.1111/1468-0009.12210

2. Siontis KC, Ioannidis JPA. Replication, duplication, and waste in a quarter million systematic reviews and meta-analyses. *Circ Cardiovasc Qual Outcomes.* 2018;11(12):e005212. doi:10.1161/CIRCOUTCOMES.118.005212

3. Riaz IB, Khan MS, Riaz H, Goldberg RJ. Disorganized systematic reviews and meta-analyses: time to systematize the conduct and publication of these study overviews? *Am J Med.* 2016;129(3):339.e11-339.e18. doi:10.1016/j.amjmed.2015.10.009

4. Bolland MJ, Grey A. A case study of discordant overlapping meta-analyses: vitamin D supplements and fracture. *PLoS ONE.* 2014;9(12):e115934. doi:10.1371/journal.pone.0115934

5. Ker K, Roberts I. Exploring redundant research into the effect of tranexamic acid on surgical bleeding: further analysis of a systematic review of randomised controlled trials. *BMJ Open.* 2015;5(8):e009460. doi:10.1136/bmjopen-2015-009460

6. Sigurdson MK, Khoury MJ, Ioannidis JPA. Redundant meta-analyses are common in genetic epidemiology. *J Clin Epidemiol.* 2020;127:40-48. doi:10.1016/j.jclinepi.2020.05.035

7. Zhang H, Tang H, He Q, et al. Surgical versus conservative intervention for acute achilles tendon rupture: a PRISMA-compliant systematic review of overlapping meta-analyses. *Medicine (Baltimore).* 2015;94(45):e1951. doi:10.1097/MD.0000000000001951

8. McDonald S, Turner S, Page MJ, Turner T. Most published systematic reviews of remdesivir for COVID-19 were redundant and lacked currency. *J Clin Epidemiol.* 2022;146:22-31. doi:10.1016/j.jclinepi.2022.02.006

9. Bolland MJ, Grey A, Reid IR. Differences in overlapping meta-analyses of vitamin D supplements and falls. *J Clin Endocrinol Metab.* 2014;99(11):4265-4272. doi:10.1210/jc.2014-2562

10. Campbell KA, Erickson BJ, Saltzman BM, et al. Is local viscosuplementation injection clinically superior to other therapies in the treatment of osteoarthritis of the knee: a systematic review of overlapping meta-analyses. *Arthrosc J Arthrosc Relat Surg.* 2015;31(10):2036-2045.e14. doi:10.1016/j.arthro.2015.03.030
11. Capodanno D. Overlapping meta-analyses of bioresorbable vascular scaffolds versus everolimus-eluting stents: bringing clarity or confusion? *J Thorac Dis*. 2016;8(7):1366-1370. doi:10.21037/jtd.2016.05.14

12. Naudet F, Schuit E, Ioannidis JPA. Overlapping network meta-analyses on the same topic: survey of published studies. *Int J Epidemiol*. 2017;46(6):1999-2008. doi:10.1093/ije/dyx138

13. Doundoulakis I, Antza C, Apostolidou-Kiouti F, et al. Overview of systematic reviews of non-vitamin K oral anticoagulants in atrial fibrillation. *Circ Cardiovasc Qual Outcomes*. 2018;11(12):e004769. doi:10.1161/CIRCOUTCOMES.118.004769

14. Ioannidis JPA. Meta-analyses in environmental and occupational health. *Occup Environ Med*. 2018;75(6):443-445. doi:10.1136/oemed-2016-104128

15. Guelimi R, Afach S, Régnaux JP, et al. Overlapping network meta-analyses on psoriasis systemic treatments: an overview, quantity does not make quality. *Br J Dermatol*. 2021;187(1):29-41. doi:10.1111/bjd.20908

16. Mendoza JFW, Latorraca C d OC, Oliveira R d Á, et al. Methodological quality and redundancy of systematic reviews that compare endarterectomy versus stenting for carotid stenosis. *BMJ Evid-Based Med*. 2021;26(1):14-18. doi:10.1136/bmjebm-2018-111151

17. Ioannidis JPA. Why most discovered true associations are inflated. *Epidemiology*. 2008;19(5):640-648. doi:10.1097/EDE.0b013e31818131e7

18. El Bahri M, Wang X, Biaggi T, Falissard B, Naudet F, Barry C. A multiverse analysis of meta-analyses assessing acupuncture efficacy for smoking cessation evidenced vibration of effects. *J Clin Epidemiol*. 2022;152:140-150. doi:10.1016/j.jclinepi.2022.09.001

19. Palpacuer C, Hammas K, Duprez R, Laviolle B, Ioannidis JPA, Naudet F. Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform

an indirect comparison meta-analysis. *BMC Med*. 2019;17(1):174. doi:10.1186/s12916-019-1409-3

20. Patel CJ, Burford B, Ioannidis JPA. Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. *J Clin Epidemiol*. 2015;68(9):1046-1058. doi:10.1016/j.jclinepi.2015.05.029

21. Klau S, Hoffmann S, Patel CJ, Ioannidis JP, Boulesteix AL. Examining the robustness of observational associations to model, measurement and sampling uncertainty with the vibration of effects framework. *Int J Epidemiol*. 2021;50(1):266-278. doi:10.1093/ije/dyaa164

22. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. *J Chiropr Med*. 2016;15(2):155-163. doi:10.1016/j.jcm.2016.02.012

23. Stan Development Team. RStan: the R interface to Stan. 2022 https://mc-stan.org/

## SUPPORTING INFORMATION

Additional supporting information can be found online in the Supporting Information section at the end of this article.

**How to cite this article:** Chapelle C, Le Teuff G, Zufferey PJ, Laporte S, Ollier E. A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism. *Res Syn Meth*. 2023;1-13. doi:10.1002/jrsm.1676
11. Capodanno D. Overlapping meta-analyses of bioresorbable vascular scaffolds versus everolimus-eluting stents: bringing clarity or confusion? *J Thorac Dis*. 2016;8(7):1366-1370. doi:10.21037/jtd.2016.05.14

12. Naudet F, Schuit E, Ioannidis JPA. Overlapping network meta-analyses on the same topic: survey of published studies. *Int J Epidemiol*. 2017;46(6):1999-2008. doi:10.1093/ije/dyx138

13. Doundoulakis I, Antza C, Apostolidou-Kiouti F, et al. Overview of systematic reviews of non-vitamin K oral anticoagulants in atrial fibrillation. *Circ Cardiovasc Qual Outcomes*. 2018;11(12):e004769. doi:10.1161/CIRCOUTCOMES.118.004769

14. Ioannidis JPA. Meta-analyses in environmental and occupational health. *Occup Environ Med*. 2018;75(6):443-445. doi:10.1136/oemed-2016-104128

15. Guelimi R, Afach S, Régnaux JP, et al. Overlapping network meta-analyses on psoriasis systemic treatments: an overview, quantity does not make quality. *Br J Dermatol*. 2021;187(1):29-41. doi:10.1111/bjd.20908

16. Mendoza JFW, Latorraca C d OC, Oliveira R d Á, et al. Methodological quality and redundancy of systematic reviews that compare endarterectomy versus stenting for carotid stenosis. *BMJ Evid-Based Med*. 2021;26(1):14-18. doi:10.1136/bmjebm-2018-111151

17. Ioannidis JPA. Why most discovered true associations are inflated. *Epidemiology*. 2008;19(5):640-648. doi:10.1097/EDE.0b013e31818131e7

18. El Bahri M, Wang X, Biaggi T, Falissard B, Naudet F, Barry C. A multiverse analysis of meta-analyses assessing acupuncture efficacy for smoking cessation evidenced vibration of effects. *J Clin Epidemiol*. 2022;152:140-150. doi:10.1016/j.jclinepi.2022.09.001

19. Palpacuer C, Hammas K, Duprez R, Laviolle B, Ioannidis JPA, Naudet F. Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform

an indirect comparison meta-analysis. *BMC Med*. 2019;17(1):174. doi:10.1186/s12916-019-1409-3

20. Patel CJ, Burford B, Ioannidis JPA. Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. *J Clin Epidemiol*. 2015;68(9):1046-1058. doi:10.1016/j.jclinepi.2015.05.029

21. Klau S, Hoffmann S, Patel CJ, Ioannidis JP, Boulesteix AL. Examining the robustness of observational associations to model, measurement and sampling uncertainty with the vibration of effects framework. *Int J Epidemiol*. 2021;50(1):266-278. doi:10.1093/ije/dyaa164

22. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. *J Chiropr Med*. 2016;15(2):155-163. doi:10.1016/j.jcm.2016.02.012

23. Stan Development Team. RStan: the R interface to Stan. 2022 https://mc-stan.org/

## SUPPORTING INFORMATION

Additional supporting information can be found online in the Supporting Information section at the end of this article.

**How to cite this article:** Chapelle C, Le Teuff G, Zufferey PJ, Laporte S, Ollier E. A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism. *Res Syn Meth*. 2023;1-13. doi:10.1002/jrsm.1676