Package: langroom
A minimal environment to evaluate embodied question answering in interactive agents.In LangRoom, agents must learn to both move and talk. LangRoom contains four objects with randomly generated colors. Agents have a partial view of the environment and receives questions "what color is ?". In response, they must seek out the correct object and generate the right answer.💬 Getting StartedPlay as a human:pipinstalllangroom# Move with WASD# Speak tokens 1-10 with number keys 1234567890pythonrun_gui.pyCreate an environment instance:importlangroomenv=langroom.LangRoom()ac={"move":0,"talk":0,"reset":True}obs=env.step(ac)📑 DocumentationTask StructureThere are four objects with fixed positions and randomized colors. By default, the agent has a partially observed view of 5x5 grid cells and cannot see all the objects at once. The environment generates questions "what color is<object>?" and then waits ten timesteps before starting to say "it is<color>". Agents answer correctly if they output the correct<color>token at the same timestep as the environment. After each question-answer sequence, the colors of the objects are re-randomized.Three reward variants are implemented, specified by thetaskargument:answer-only: agent is rewarded only for saying the correct color at the right timestep and penalized a small amount for saying things at other timesteps. Use this reward structure for comparability to the original paper Lin et al. (2023).answer-and-echo: agent is rewarded for predicting tokens that the environment generates (including silences and questions), with a larger reward for saying the correct color at the right timestep.echo: agent is rewarded for predicting all tokens the environment generates equally.Observation SpaceThe observation and action space definition follows theembodiedenvironment interface.image (uint8 (resolution, resolution, 3)): pixel agent-centric local viewtext (uint32 ()): ID of the token at the current timesteplog_image (uint8 (resolution, 4 * resolution, 3)): debugging view with additional information rendered with agent viewFollowing theembodiedenv interface, these keys are also provided in the observation:reward (float32): reward at the current timestepis_first (bool): True if this timestep is the first timestep of an episodeis_last (bool): True if this timestep is the last timestep of an episode (terminated or truncated)is_terminal (bool): True if this timestep is the last timestep of an episode (terminated)Action SpaceLangRoom has a dictionary action space that allows the agent to output actions and tokens (i.e. move and speak) simultaneously at each timestep.move (int32 ()): ID of the movement action from movement action space[stay down up right left]talk (int32 ()): ID of the generated tokenFollowing theembodiedenv interface, the action space also includes:reset (bool): set to True to reset the episodeVocabulary SizeBy default, the vocabulary size is 15 (the minimal number of tokens to ask and answer questions). To test how agents deal with larger vocabularies (and thus larger action spaces), set thevocab_sizeargument. Additional words in the vocabulary will be filled with dummy tokens.🛠️ Development and IssuesNew development and extensions to the environment are welcome! For any questions or issues, please open a GitHub issue.Citation@article{lin2023learning,
         title={Learning to Model the World with Language},
         author={Jessy Lin and Yuqing Du and Olivia Watkins and Danijar Hafner and Pieter Abbeel and Dan Klein and Anca Dragan},
         year={2023},
         eprint={2308.01399},
         archivePrefix={arXiv},
}