DOI: 10.5281/zenodo.16526642
title: Dataset for: Metamorphic Testing of Large Language Models for Natural Language Processing (ICSME'25)

description: # Dataset for: Metamorphic Testing of Large Language Models for Natural Language Processing (ICSME'25)

## Overview

This repository contains the data associated with the paper [Metamorphic Testing of Large Language Models using Natural Language Processing](https://valerio-terragni.github.io/assets/pdf/cho-icsme-2025.pdf), accepted at ICSME 2025. The associated tool, LLMorph, can be found at: https://github.com/steven-b-cho/llmorph

Authors: Steven Cho, Stefano Ruberto, Valerio Terragni

If you use this dataset, please cite:

```@inproceedings{cho2025metamorphic,  author = {Cho, Steven and Ruberto, Stefano and Terragni, Valerio},  title = {Metamorphic Testing of Large Language Models for Natural Language Processing},  booktitle = {Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME)},  year = {2025},  publisher = {IEEE}}```

## Description

All data is given as JSON files.

```mt4llm-data/├── rq1-failure-rate/results/├── rq2-ground-truth-comparison/│   ├── source-inputs-with-labels/│   ├── source-output-correctness/│   └── ground_truth_comparison_summary.json├── rq3-manual-validation/manually_labelled_violations.json├── rqextra-flakiness/│   ├── results/│   ├── fail_rates.json│   ├── fail_rates_summarised.json│   └── manual_vs_flakiness_comparison.json├── summary/│   ├── data_summary_by_category.json│   └── data_summary_by_cid.json└── README.md```

### RQ1

`rq1/results` contains the raw output of the data for testing relations on tasks on LLMs. The properties are:

- `llm_name`, `task_name`, `relation_name`: The used LLM, task, and MR, respectively.- `data`: The list of data instances.    - `id`: The nth instance in this file.    - `source_input`, `source_output`, `followup_input`, `followup_output`: The source and follow-up inputs and outputs. Note that some cases may produce multiple follow-up inputs and outputs.    - `relations`: Whether the output relation of the MR was satisfied for each follow-up output.    - `verif_failure`: Whether any additional constraints were violated. `null` if all satisfied.

### RQ2

`rq2` compares the the metamorphic data with ground truths.

`rq2/source-inputs-with-labels`: Contains the original test datasets used for the experiments (with ground truth labels).

`rq2/source-output-correctness`: Contains lists of boolean values on whether the source outputs for a particular LLM-task pair aligns with the corresponding ground truths from the original test datasets.

`rq2/ground_truth_comparison_summary.json`: A summary of the ground truth comparisons for each LLM-task-relation tuple. Each entry contains:

- `llm_name`, `task_name`, `relation_name`: The target LLM, task, and MR, respectively.- `total`: The total number of valid metamorphic test cases run.- `mr_violated`: The total number of cases with metamorphic violations.- `mr_satisfied_gt_correct`, `mr_satisfied_gt_incorrect`, `mr_violated_gt_correct`, `mr_violated_gt_incorrect`: The number of cases where the MR was satisfied/violated and correct/incorrect according to the ground truth.

### RQ3

`rq3/manually_labelled_violations.json`: A collection of metamorphic violations, manually labelled as to the type of violation. The properties are:

- `source_file`: The file name in `rq1/results` where the instance appears.- `llm_name`, `task_name`, `relation_name`: The used LLM, task, and MR, respectively.- `id`: The id of this instance found in the above `source_file`.- `source_input`, `source_output`, `followup_input`, `followup_output`: The source and follow-up inputs and outputs.- `output_relation`: Whether the output relation of the MR was satisfied. Always `false` in this case.- `followup_index`: Some tests produce multiple follow-up inputs/outputs. Here, each individual input-output pair is regarded separately. This value indicated the nth follow-up input/output used as found in the `source_file`.- `true_output`: The ground truth, as found in the original test datasets.- `failure_type`: The manually labelled type of violation. Values are `0` (true failure), `1` (intput transformation failure), `2` (output comparison failure), `3` (failure due to nonsensical MR), `4` (other failures), `5` (failure to identify 'unknown' in QA), and `6` (failure to recognise equivalence in RE).

### RQE

`rqe` looks at re-running failing metamorphic cases 9 extra times to test for flakiness.

`rqe/results`: Contains the raw data of 10 runs of test cases. Each instance has:

- `source_file`: The file name in `rq1/results` where the instance appears.- `llm_name`, `task_name`, `relation_name`: The used LLM, task, and MR, respectively.- `id`: The id of this instance found in the above `source_file`.- `followup_index`: Some tests produce multiple follow-up inputs/outputs. Here, each individual input-output pair is regarded separately. This value indicated the nth follow-up input/output used as found in the `source_file`.- `source_input`, `followup_input`: The source and follow-up inputs.- `data`: The list of 10 instances run.    - `followup_input`, `followup_output`: The source and follow-up outputs.    - `output_relation`: Whether the outponal constraints were violated. `null` if all satisfied.

`rqe/fail_rates.json`: Contains the calcaulated metamorphic violation rate for each run as described above, grouped by LLM-task-MR tuple.

`rqe/fail_rates_summarised.json`: Contains the frequency of runs with violation rates of `0%` to `100%`, by `10%` increments. Grouped by LLM-task-MR tuple.

`manual_vs_flakiness_comparison.json`: Takes the instances with manually labelled violation types (from `rq3/manually_labelled_violations.json`) and joins with the re-run violation rates (from `rqe/fail_rates.json`).

### Summary

`summary` contains the summary data for the experiments.

`summary/data_summary_by_cid.json`: Contains various summarised properties, grouped by LLM-task-MR tuple. Each instance contains:

- `llm_name`, `task_name`, `relation_name`: The target LLM, task, and MR, respectively.- `total_count`: The total number of valid metamorphic test cases run.- `fail_count`: The total number of cases with metamorphic violations.- `fail_rate`: The proportion of cases that failed.- `fail_type`: The number of manually analysed failing cases for each respecive fail type. See `rq3` above for details.- `rerun_counts`: The number of failing cases for each flakiness rate. `null` for GPT-4, as those cases were not re-run. See `rqe` above for details.- `ground_truth_comparison`: The number of cases where the MR was satisfied/violated and correct/incorrect according to the ground truth. See `rq2` above for details.

`summary/data_summary_by_category.json`: Contains various properties as described above (except `rerun_counts` and `ground_truth_comparison`), summarised individually by LLM, task, and MR.