Discussion on a null hypothesis for a bibliometric study
Authors: Kraus Corinne , MosgÃ¶ller Wilhelm , Lothaller Harald , Endler P Christian 
Published: July 2021

In a bibliometric study [1] we located around 100 high-potency studies, each of which scrutinized an earlier study. An expert round was set to discuss whether it was justified to use the null hypothesis that no difference would be found between the respective test and control groups.

The following two viewpoints were juxtaposed:

1) The classical null hypothesis states that the results from a test group and its respective control will not differ. If difference is found, the null hypothesis must be rejected. In this case, whether the values found in the test group were higher or lower than the ones of the control group is irrelevant. However, in some instances lack of difference between the test and the control groups is not helpful to make an informed decision. In the case of the analyzed bibliometric study, the research question is as follows: is the number of replication studies which produced a result of the same kind as the original study greater than the number of replication studies which produced a result of the opposite kind?

2) Often there is a pragmatic interest in a one-sided difference or in a deviation in one specific direction. A null hypothesis to this end will state, for example, that the result of the test group will not be better than that of the control group. If difference in favor of the test group is found, the null hypothesis must then be rejected. If, conversely, no difference is found or the result of the control group is better, the null hypothesis can be maintained. For the analyzed bibliometric study the question is: is the number of replication studies which yielded a result of the same kind as the original study greater than the number of all other replication studies?

From viewpoint (1) the outcome was that 70% of replication studies yielded the same result as the original study, 10% per cent the opposite result, and in 20% no difference was found between test and control. From this view point the standing is 70 positive, 10 negative and 20 undecided results. Broken down further, the outcome for laboratory-internal replication was 83 and 5 (12 undecided results) while the outcome for multicenter replication was 75 and 11 (14) and that for external replication was 48 and 14 (38).

From viewpoint (2) the outcome was that 70% of replication studies yielded the same result as the original study, and that 30% (= 10% + 20%) either resulted in no difference between the test and the control group or a result opposite to the one of the original study. From this viewpoint the standing is 70 â€˜for reproducibilityâ€™ versus 30 â€˜not for reproducibility.â€™ Broken down further, the outcome for laboratory-internal replication was 83 and 17 (= 5 + 12), while that for multicenter replication 75 and 25 (= 11 + 14) and the one for external replication 48 and 52 (= 14 + 38).

We conclude that the 2 two viewpoints should be treated as equally valid when discussing the outcome of the analyzed bibliometric study

Keywords: high dilution, experiment repetition, null hypothesis

References
[1] Endler PC, Bellavite P, Bonamin L, JÃ¤ger T. Mazon S. Replications of fundamental research models in ultra high dilutions, 1994 and 2015: an update on a bibliometric study. Homeopathy 2015;104:235-245