# Exploring the bibliometric and semantic nature of negative

Dominique Besagni, Claire François, Juan Gorraiz, Christian Gumpenberger, Ivana Roche, Edgar L. Schiebel, Martin Wieland

► To cite this version:

Dominique Besagni, Claire François, Juan Gorraiz, Christian Gumpenberger, Ivana Roche, et al. "Exploring the bibliometric and semantic nature of negative. 11th International Conference on Science & Technology Indicators - STI'2010, Sep 2010, Leiden, Netherlands. pp.37-40. hal-00614062

HAL Id: hal-00614062

https://hal.science/hal-00614062v1

Submitted on 9 Aug 2011

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Exploring the bibliometric and semantic nature of negative results

Dominique Besagni¹, Claire François¹, Juan Gorraiz², Christian Gumpenberger², Ivana Roche¹,
Edgar Schiebel³, Martin Wieland²

¹dominique.besagni, claire.francois, ivana.roche@inist.fr
INIST-CNRS Institut de l'Information Scientifique et Technique (France),

²juan.gorraiz, christian.gumpenberger, martin.wieland@univie.ac.at
University of Vienna (Austria)

³edgar.schiebel@ait.ac.at
Austrian Institute of Technology (Austria)

**Keywords:** bibliometrics, scientometrics, negative results, negative result publication, S&T information

## Background

Scientific progress in some disciplines is hampered by researchers' tendencies to bin negative results. However, their publication is beneficial in order to prevent duplication of effort, save public money and to facilitate and promote scientific communication and progress.

A bibliometric analysis of negative results literature was performed in order to identify their most important attributes, to study their outcome and to explore their hidden relationships, focussing on the distribution of publications openly declared as containing negative results and published in journals completely devoted to this kind of publications.

The Journal of Negative Results in BioMedicine (JNRBM) was used as a role model, since all the other relevant journals are not indexed in the largest citation databases.

In order to evaluate the prestige of this journal, the impact factor as well as the new alternative indicators SJR and SNIP were looked up for all assigned categories.

The complete article records of JNRBM, the number of citations and the citation percentiles & averages were retrieved from Scopus and Web of Knowledge to generate an impact profile.

Data analysis and visualization of relationships were performed using the software tool BibTechMon. A co-author and a co-affiliation map were produced in order to examine the plurality of communities based on the existing relationships between the authors publishing in JNRBM.

Furthermore two samples of "positive results" based on related articles containing the same number of items, either sharing the same descriptors or the same references (bibliographical coupling) were compiled in Scopus.
The first sample, obtained from Scopus by retrieving articles sharing the maximal number of references, was used to test if bibliographic coupling revealed other publications of negative results.

The second sample of “positive results”, sharing the maximal number of descriptors, was used to enable a comparison with the sample of “negative results”. Thereafter, their abstracts were subjected to assisted data extraction, in order to identify vocabulary characterising negative results publications.

## Results

The unofficial impact factor of JNRBM is 1.64 based on the information gained from the official journal website. The median impact factor (IF) in the corresponding WoS category “Medicine, research & experimental” (to which biomedical science journals are assigned) is 2.023, and the aggregate IF is 3.474. Thus the IF of JNRBM is below these values and corresponds to quartile 3 (Q3). According to Scopus data, in SCImago Journal Rank JNRBM has a SJR of 0.124 and is assigned to the categories “Medicine” (Q1) and “Pharmacology, Toxicology and Pharmaceutics” (Q2), whereas in CWTS Journal Indicators a SNIP of 0.43 is reached in the category “Biochemistry, Genetics and Molecular Biology (all)” (Q2).

JNRBM publications (93% of them articles) are cited by a broad spectrum of journals rather than by specific titles. Journals exclusively devoted to their official publication like JNRBM have a rather low impact. Only 11% of their items are above citation average. However, only one third of the publications remain uncited.

<figure><img src="image_2.png" /><figcaption>**Figure 1.** Average citations of publications in JNRBM (2002-2009)</figcaption></figure>

Apart from single publications there is no specific community for the publication of negative results in devoted literature like JNRBM. Authorship is widely spread, with 95% of the authors contributing with a unique paper and the most active author publishing 6 papers.
Neither the co-author nor the co-affiliation analyses indicate a strong interconnectivity of authors or affiliations. There are only few clusters, whereas the majority of the publications have affiliations originating from a single country.

<figure><img src="image_3.png" /><figcaption>Fig.2a: Co-authors map, Articles: 75, authors: 377, edges: 1218; network density: 0.01718; source: Scopus, 2009, 11., *Journal of Negative Results in BioMedicine*</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig.2b: Co-authors map, Articles: 75, authors: 528, edges: 4352, network density: 0.03128; source: Scopus, 2010, 2., *same descriptors as JNRBM*.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig.2c: Co-Affiliations map, Articles: 75, affiliations: 213, edges: 371; network density: 0,01643; source: Scopus, 2009, 11., *Journal of Negative Results in BioMedicine*</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig.2d: Co-Affiliations map, Articles: 75, affiliations: 237, edges: 1253; network density: 0,0448; source: Scopus, 2010, 2., *same descriptors as JNRBM*.</figcaption></figure>

**Figure 2. Co-authorships and co affiliations of publications in JNRBM (2002-2009) and similar publications (high concordance in descriptors) in Scopus**
Our first analysis shows that bibliographic coupling can be useful for the identification of other
negative results, however, can only be regarded as a supplementary method.

A text mining approach was applied to the JNRBM publications and resulted in a list of around
50 terms expressing negative assertions. These were employed to query the database PASCAL.
An indicator measuring the degree of negativeness of a publication is defined. This indicator can
be then applied to both the PASCAL corpus and the JNRBM publications and the obtained
results compared.

At this stage it is impossible to distinguish between the various possible negation meanings. This
exploratory work opened a challenge in the refinement of mining techniques - not only to
detect negative assertions, but especially to allow categorization by their real semantic
meaning.

## Acknowledgments

We express our gratitude to Mr. Abel L. Packer, Director of BIREME/OPS/OMS for his inspiring
suggestions and helpful feedback.

## References

- Browman H.I. (1999), *Negative results*, Marine Ecology Progress Series 191, 301-309.

- Dickersin K. (1990), *The existence of publication bias and risk factors for its occurrence*, JAMA 263, 10, 1385-1389.

- Kopcsa A. and Schiebel E. (2001), *Content Visualisation by BibTechMon™*, In: Informatics 2001/Network Economy-Visions and Reality, OCG-Austrian Computer Society

- Valette M. (2004), *Sémantique interprétative appliquée à la detection automatique de documents racistes et xénophobes sur internet*, Approches Sémantiques du Document Numérique, Actes du 7e Colloque International sur le Document Electronique, 22-25 juin 2004, Patrice Enjalbert et Mauro Gaio, ed., 2004, pp. 215-230