DOI: 10.57761/z335-9m16
title: GazexPErT

description: Corresponding authors: Joy T. Wu (joytywu@stanford.edu) and Daniel Beckmann (d_beck26@uni-muenster.de)
description: [18F]FDG-PET/CT is a cornerstone imaging modality for evaluating tumor staging and treatment response across many cancer types, yet expert reader shortages necessitate more efficient diagnostic aids. While standalone AI models for automatic lesion detection exist, clinical translation remains hindered by concerns about explainability, reliability, and workflow integration. We present GazeXPErT, a 4D eye-tracking dataset with expert decisions for tumor detection and measurement on 346 dual-annotated FDG-PET/CTs. The dataset contributes 9,030 unique gaze-to-lesion trajectories derived from 3,948 minutes of 60Hz eye-tracking data, rendered in COCO-style format for machine learning applications. GazeXPErT captures experts' visual reasoning patterns when detecting suspicious malignant lesions. It aims to facilitate development of explainable AI models and enable human-AI interaction research through understanding expert gaze patterns. Baseline validation experiments demonstrate that a 3D nnUNet segmentation model achieves superior performance when utilizing expert gaze patterns versus without (DICE score 0.6819 versus 0.6008), and that sequential gaze and PET/CT images can be used to improve lesion localization and expert intention prediction.

GazeXPErT is designed to explore multiple machine learning problems for FDG-PET/CTs beyond the baseline experiments shown in our paper, including and not limited to, visual grounding or causal reasoning, clinically explainable feature augmentation, human-computer interaction, human intention prediction or understanding, and expert gaze-rewarded modeling.
description: **Overview: **

GazexPErT is an eye-tracking dataset with dual annotated radiology expert eye-tracking data for tumor lesion detection and measurement on 346 FDG-PET/CT studies with positive tumors, which were sampled from an open source Tubingen Whole Body FDG-PET/CT dataset [1]. The GazeXEPrT dataset captures the human experts' search patterns for the task of differentiating and detecting hypermetabolic lesions in one of the most common cancer staging imaging modality. This dataset aims to facilitate the development of AI tools/models that can read FDG-PET/CT scans with expert insight and/or interact with human experts through understanding their gaze patterns.



**Data collection: **

We built an interactive PET/CT read platform that integrated a screen-based research eye-tracker (60Hz) with a 27-inch diagnostic monitor, a baseline lesion segmentation algorithm, and a custom user interface built on Python (UI). The lesion-segmentation algorithm generates tumor proposals via expert adjustable SUV thresholding and connected components analysis, prioritizing speed and sensitivity for lesion-level annotation purposes. Using this purpose-built research eye-tracking platform, gaze coordinates and all image metadata are collected in real-time at 60Hz as the radiologists freely read the [18F] FDG-PET/CTs in axial and MIP views. The UI allows "gaze selection and measurement" of abnormal FDG-avid foci. The readers can resize, accept, or reject a lesion proposed by the baseline segmentation algorithm based on their gaze location. The overall annotation goal is to gaze-select and validate all active tumors, creating the maximum number of gaze-to-lesion search pattern trajectories for training downstream interpretable and explainable machine learning applications. Note that report creation was not within the scope of version 1 of the GazePErT dataset. However, we plan to continue to expand on this dataset to additional FDG-PET/CTs from Stanford and other public sources.





**Reference:**

[1] Gatidis S, Hepp T, Früh M, La Fougère C, Nikolaou K, Pfannenberg C, Schölkopf B, Küstner T, Cyran C, Rubin D. A whole-body FDG-PET/CT dataset with manually annotated tumor lesions. Scientific Data. 2022 Oct 4;9(1):601.




description: **Dataset high level statistics: **

346 FDG-PET/CT studies (98 lymphoma, 122 lung cancer, and 126 melanoma cases), each read by 1 trainee and 1 board certified specialist in nuclear medicine or diagnostic radiology. Each PET/CT read took 2-5 minutes on average and all sequential data needed to reconstruct the whole read as a video recording is saved in a JSON data format, separate from the source PET/CT images. After post-processing the 3,948 minutes of raw eye-tracking data, GazeXPErT contributes 9,030 unique gaze-to-lesion trajectories (60Hz) rendered in a COCO-style format, friendly for multiple downstream machine learning tasks. Please see our detailed dataset paper, which describes the data collection process, data schema, postprocessing and example baseline technical validation experiments.