Question: What tasks does the general platform expand?
Evidence: Human learning occurs through interaction BIBREF0 and multimodal experience BIBREF1 , BIBREF2 . Prior work has argued that machine learning may also benefit from interactive, multimodal learning BIBREF3 , BIBREF4 , BIBREF5 , termed virtual embodiment BIBREF6 . Driven by breakthroughs in static, unimodal tasks such as image classification BIBREF7 and language processing BIBREF8 , machine learning has moved in this direction. Recent tasks such as visual question answering BIBREF9 , image captioning BIBREF10 , and audio-video classification BIBREF11 make steps towards learning from multiple modalities but lack the dynamic, responsive signal from exploratory learning. Modern, challenging tasks incorporating interaction, such as Atari BIBREF12 and Go BIBREF13 , push agents to learn complex strategies through trial-and-error but miss information-rich connections across vision, language, sounds, and actions. To remedy these shortcomings, subsequent work introduces tasks that are both multimodal and interactive, successfully training virtually embodied agents that, for example, ground language in actions and visual percepts in 3D worlds BIBREF3 , BIBREF4 , BIBREF14 .
For virtual embodiment to reach its full potential, though, agents should be immersed in a rich, lifelike context as humans are. Agents may then learn to ground concepts not only in various modalities but also in relationships to other concepts, i.e. that forks are often in kitchens, which are near living rooms, which contain sofas, etc. Humans learn by concept-to-concept association, as shown in child learning psychology BIBREF1 , BIBREF2 , cognitive science BIBREF15 , neuroscience BIBREF16 , and linguistics BIBREF17 . Even in machine learning, contextual information has given rise to effective word representations BIBREF8 , improvements in recommendation systems BIBREF18 , and increased reward quality in robotics BIBREF19 . Importantly, scale in data has proven key in algorithms learning from context BIBREF8 and in general BIBREF20 , BIBREF21 , BIBREF22 .
To this end, we present HoME: the Household Multimodal Environment (Figure 1 ). HoME is a large-scale platform for agents to navigate and interact within over 45,000 hand-designed houses from the SUNCG dataset BIBREF23 . Specifically, HoME provides:
HoME is a general platform extensible to many specific tasks, from reinforcement learning to language grounding to blind navigation, in a real-world context. HoME is also the first major interactive platform to support high fidelity audio, allowing researchers to better experiment across modalities and develop new tasks. While HoME is not the first platform to provide realistic context, we show in following sections that HoME provides a more large-scale and multimodal testbed than existing environments, making it more conducive to virtually embodied learning in many scenarios.
Specifically, HoME provides: HoME is a general platform extensible to many specific tasks, from reinforcement learning to language grounding to blind navigation, in a real-world context.
Answer: The general platform could extense to many specific tasks, from reinforcement learning to language grounding to blind navigation.