On Non-interactive Evaluation of Animal Communication Translators
Authors: name: Orr Paradise, name: David F. Gruber, name: Adam Tauman Kalai

If you had an AI Whale-to-English translator, how could you validate whether
or not it is working? Does one need to interact with the animals or rely on
grounded observations such as temperature? We provide theoretical and
proof-of-concept experimental evidence suggesting that interaction and even
observations may not be necessary for sufficiently complex languages. One may
be able to evaluate translators solely by their English outputs, offering
potential advantages in terms of safety, ethics, and cost. This is an instance
of machine translation quality evaluation (MTQE) without any reference
translations available. A key challenge is identifying ``hallucinations,''
false translations which may appear fluent and plausible. We propose using
segment-by-segment translation together with the classic NLP shuffle test to
evaluate translators. The idea is to translate animal communication, turn by
turn, and evaluate how often the resulting translations make more sense in
order than permuted. Proof-of-concept experiments on data-scarce human
languages and constructed languages demonstrate the potential utility of this
evaluation methodology. These human-language experiments serve solely to
validate our reference-free metric under data scarcity. It is found to
correlate highly with a standard evaluation based on reference translations,
which are available in our experiments. We also perform a theoretical analysis
suggesting that interaction may not be necessary nor efficient in the early
stages of learning to translate.