DOI: 10.5281/zenodo.11509228
title: Best practices for evaluating IRFL approaches - Supplemental material

description: Information retrieval fault localization (IRFL) is a popular research field and many IRFL approaches have been proposed recently.Unfortunately, the evaluation of some of these IRFL approaches is often too simplistic, which can cause an overestimation of performance of these approaches.For this reason, we discuss evaluation pitfalls and problems in our paper "Best practices for evaluating IRFL approaches".To support our claim of the poor status quo of current evaluation practices in some research papers, we have performed a literature survey on 135 papers.This repository contains the data that we have extracted from these papers and our evaluation script.
description: Raw data:  scopus search 26.9.2023.csv (initial search) and scopus search 22.10.2024.csv (updated search)

Evaluation script: evaluation.py

Requirements: requirements.txt
description: The list of investigated papers in scopus search 22.10.2024.csv is based on a Scopus search performed on 2024-10-22 with the following query:

SUBJAREA ( comp ) AND ( TITLE-ABS-KEY ( "information retrieval" ) OR TITLE-ABS-KEY ( "IR" ) ) AND ( TITLE-ABS-KEY ( "fault localization" ) OR TITLE-ABS-KEY ( "bug localization" ) )
description: Content of the raw data

Researcher 1 extracted the information from the papers and Researcher 2 verified it. We used the papers as the only source of information, and we did not try to obtain missing information from supplemental online material, because only a small fraction of the papers links to a replication package.



bib: a unique key to identify this paper

is_paper: 1 (if it is a research paper), 0 (otherwise)

is_relevant: 1 (if it is within the scope of our survey), 0 (otherwise)

reason for exclusion: empty for relevant papers, otherwise: textual description of exclusion reason

metrics: list of metrics that have been used in the evaluation section of this paper, separated by comma

evaluation strategy: e.g., bootstrap, one off experiment, cross validation

train test split: e.g., no, 'train, test split', 'timeseries cross validation'

misuse of time-related issue tracking data (according to Tu2018): definitely, potentially, empty

statistical tests performed: names of the performed tests (including post-hoc tests), separated by comma

Posthoc required: empty (when not neccessary), yes (when used), should have been used

significance claimed: yes, no

significance threshold reported: yes, no, 'no, but p values reported'

effect size: no or name

score combination: e.g. weighted linear combination, unknown, LTR (learning to rank)

weights / hyperparameters determined: e.g., post mortem, post mortem analysis, unknown, validation set, training set

Replication package: URL or - (not provided)

content replication package: na (when no link was provided), '404 - not found', otherwise content (e.g., dataset, replication package)

Paper metadata (as obtained from scopus search)



Authors

Title

Year

Source title

Volume

Issue

Art. No.

Page start

Page end

Page count

DOI

Document Type

Open Access
description: Evaluation script

This python file contains a main as entry point. All libraries mentioned in the Requirements should be installed. Then the script can be run using the command python evaluation.py. This files creates all figures and LaTeX tables used in the paper. Additionally, it outputs the citation keys of



papers using any of the metrics 'p', 'p@k', 'r', 'r@k'

papers using post-hoc tests

papers where any statistical tests have been performed

papers where no URL to a replication package was provided

papers where the URL to the replication package does not exist anymore


Since the papers use different names for some statistical tests and metrics, we use dictionaries in our Python script to summarize metrics and statistical tests.