AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment
Authors: name: Anastasiia Ivanova, name: Eva Bakaeva, name: Zoya Volovikova, name: Alexey K. Kovalev, name: Aleksandr I. Panov

As a part of an embodied agent, Large Language Models (LLMs) are typically
used for behavior planning given natural language instructions from the user.
However, dealing with ambiguous instructions in real-world environments remains
a challenge for LLMs. Various methods for task ambiguity detection have been
proposed. However, it is difficult to compare them because they are tested on
different datasets and there is no universal benchmark. For this reason, we
propose AmbiK (Ambiguous Tasks in Kitchen Environment), the fully textual
dataset of ambiguous instructions addressed to a robot in a kitchen
environment. AmbiK was collected with the assistance of LLMs and is
human-validated. It comprises 1000 pairs of ambiguous tasks and their
unambiguous counterparts, categorized by ambiguity type (Human Preferences,
Common Sense Knowledge, Safety), with environment descriptions, clarifying
questions and answers, user intents, and task plans, for a total of 2000 tasks.
We hope that AmbiK will enable researchers to perform a unified comparison of
ambiguity detection methods. AmbiK is available at
https://github.com/cog-model/AmbiK-dataset.