DOI: 10.5281/zenodo.17350445
title: HERITRACE user testing: packages and results

description: This dataset contains the results of the evaulation and the packages used to conduct the evaluation of HERITRACE, a semantic content management system for cultural heritage institutions. The study involved 14 participants divided into two groups: end users (7 participants) and configurators (7 participants).

Dataset contains:



Testing packages: two standalone Docker-based testing packages that enable independent reproduction and extension of the usability evaluation. 

User testing results: the dataset combines quantitative metrics (SUS scores, task duration, error rates, success rates) with qualitative analysis following grounded theory methodology (open, axial, and selective coding). All data is provided in accessible formats (JSON for structured data, Markdown for textual content, PNG for visualizations), enabling reproducibility and secondary analysis.


 Testing packages

  heritrace-enduser-testing.zip - End user testing package



Self-contained testing environment for metadata editors and domain experts

Pre-configured with 100 journal articles from OpenCitations Meta

Four guided tasks: editing records, merging duplicates, version restoration, and creating new entries

Duration: 60 minutes maximum

No Semantic Web knowledge required


  heritrace-technician-testing.zip - Configurator testing package



Environment for evaluating HERITRACE configuration capabilities

Two configuration tasks: SHACL schema extension and display rule customization

Tests system configurability for institutional customization

Duration: 60 minutes maximum

Semantic Web knowledge required


User testing results: directory structure

  heritrace_user_testing_results/  ├── aggregated_analysis/  │   ├── recruitment/ - Participant recruitment distribution data and visualizations  │   ├── sus/ - Aggregated System Usability Scale (SUS) reports, individual scores, and comparative visualizations  │   └── tasks/ - Cross-participant task metrics including duration comparisons, error heatmaps, and success rates  │  ├── endusers/ - Data from 7 end user participants (1e-7e)  │   ├── transcriptions/ - Complete verbatim transcriptions of testing sessions  │   ├── written_answers/ - Responses to post-test written questionnaires  │   ├── sus/ - Individual System Usability Scale questionnaires  │   ├── task_completion/ - Structured task performance data (completion status, duration, errors)  │   ├── open_coding/ - First-level qualitative coding of transcriptions  │   ├── end_user_axial_codes.json - Second-level categorical analysis  │   ├── end_user_selective_codes.json - Third-level theoretical integration  │   └── TEMPLATE_task_completion.json - Task completion data structure template  │  ├── technicians/ - Data from 7 technician/configurator participants (1t-7t)  │   ├── transcriptions/ - Complete verbatim transcriptions of testing sessions  │   ├── written_answers/ - Responses to post-test written questionnaires  │   ├── sus/ - Individual System Usability Scale questionnaires  │   ├── task_completion/ - Structured task performance data (completion status, duration, errors)  │   ├── open_coding/ - First-level qualitative coding of transcriptions  │   ├── technician_axial_codes.json - Second-level categorical analysis  │   ├── technician_selective_codes.json - Third-level theoretical integration  │   └── TEMPLATE_task_completion.json - Task completion data structure template  │  └── templates/ - Methodological templates documenting the analytical structure      ├── TEMPLATE_axial_codes.json - Axial coding structure      ├── TEMPLATE_open_codes.json - Open coding structure      └── TEMPLATE_selective_codes.json - Selective coding structure