Sightation Counts: Leveraging Sighted User Feedback in Building a
  BLV-aligned Dataset of Diagram Descriptions
Authors: name: Wan Ju Kang; name: Eunki Kim; name: Na Min An; name: Sangryul Kim; name: Haemin Choi; name: Ki Hoon Kwak; name: James Thorne
Published: 2025-03-17 16:52:46

Abstract: Often, the needs and visual abilities differ between the annotator group and
the end user group. Generating detailed diagram descriptions for blind and
low-vision (BLV) users is one such challenging domain. Sighted annotators could
describe visuals with ease, but existing studies have shown that direct
generations by them are costly, bias-prone, and somewhat lacking by BLV
standards. In this study, we ask sighted individuals to assess -- rather than
produce -- diagram descriptions generated by vision-language models (VLM) that
have been guided with latent supervision via a multi-pass inference. The
sighted assessments prove effective and useful to professional educators who
are themselves BLV and teach visually impaired learners. We release Sightation,
a collection of diagram description datasets spanning 5k diagrams and 137k
samples for completion, preference, retrieval, question answering, and
reasoning training purposes and demonstrate their fine-tuning potential in
various downstream tasks.