What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine
  Translation with a Human-centered Study
Authors: name: Beatrice Savoldi, name: Sara Papi, name: Matteo Negri, name: Ana Guerberof, name: Luisa Bentivogli

Gender bias in machine translation (MT) is recognized as an issue that can
harm people and society. And yet, advancements in the field rarely involve
people, the final MT users, or inform how they might be impacted by biased
technologies. Current evaluations are often restricted to automatic methods,
which offer an opaque estimate of what the downstream impact of gender
disparities might be. We conduct an extensive human-centered study to examine
if and to what extent bias in MT brings harms with tangible costs, such as
quality of service gaps across women and men. To this aim, we collect
behavioral data from 90 participants, who post-edited MT outputs to ensure
correct gender translation. Across multiple datasets, languages, and types of
users, our study shows that feminine post-editing demands significantly more
technical and temporal effort, also corresponding to higher financial costs.
Existing bias measurements, however, fail to reflect the found disparities. Our
findings advocate for human-centered approaches that can inform the societal
impact of bias.