Development of an Interactive Navigation System based on
Household Ontology and Commonsense reasoning
Alan Schalkwijk

Graduate School of Science andEngineering
Aoyama Gakuin University
Sagamihara Kanagawa Japan
c5620170@aoyama.jp

Motoki Yatsu

College of Science andEngineering
Aoyama Gakuin University
Sagamihara Kanagawa Japanyatsu@it.aoyama.ac.jp

ABSTRACTIn recent years, researchers from the fields of computer vision,
language, graphics, and robotics have tackled Embodied AI
research. Embodied AI can learn through interaction with thereal world and virtual environments and can perform varioustasks in virtual environments using virtual robots. However,many of these are one-way tasks in which the interaction isinterrupted only by answering questions or requests to theuser. In this study, we aim to develop a task-oriented
interactive system using virtual household ontology andcommonsense reasoning, in which a virtual robot can reason
about the location of a guide while interacting with the user andguide the user around the house.

CCS CONCEPTS• Information systems → Information systems applications.

KEYWORDS
Navigation System, Ontology, Dialog System, Commonsense
Reasoning

1 Introduction
In recent years, research has been conducted on Embodied AI[8], which attempts to solve tasks such as object navigation andquestion answering in virtual spaces by using agents such asvirtual robots that learn in the real world and virtual spaces toguide the user to the desired object or room. In order to solvethese tasks, it is necessary to integrate technologies fromvarious fields such as computer vision, computer graphics,
natural language processing, artificial intelligence, and robotics.
Traditional research in computer vision and naturallanguage processing has been developed by 'Internet AI', which
focuses on pattern recognition of images and text. Embodied AI
focuses on the ability of a virtual robot to see, hear, speak, move,etc. in a virtual space.
In conventional Embodied AI research, tasks such as object
navigation and question answering can be solved by a one-way
dialogue in which the virtual robot simply responds to specificrequests from the user. In order for the virtual robot to be able
to respond to ambiguous requests from the user, it is necessary

Takeshi Morita

College of Science andEngineering
Aoyama Gakuin University
Sagamihara Kanagawa Japan
morita@it.aoyama.ac.jp

for the virtual robot to interact multiple times with the user andinfer his intentions.
In this research, we aim to develop a two-way interactive
navigation system by introducing knowledge inferencetechniques to Embodied AI research. Specifically, based on thehome environment ontology and common-sense reasoning, we
attempt to solve a task in which a virtual robot can reason
about the user's intentions while interacting with the user
multiple times, and guide the user to the location in the virtual
home environment that the user needs.

2 Related research
2.1 Vision and Dialog Navigation
Vision and Dialog Navigation (VDN)[1] is a task in which ahuman gives unspecified instructions to a virtual robot, such as
in a home environment, and the robot responds and guides thehuman. The VDN aims at learning a virtual robot that can
automatically guide a human in an unfamiliar place, such as averbal tele-operated home robot or an office robot, which
would await the user to give clues for predicting the facility inneed." The VDNs are trained and simulated using a realistic
simulation environment called Matterport Room2Room[11],by creating a dataset of dialogue examples and the virtual
robot's actions (forward and turn) in response to the dialogue
examples.
The main dialogue examples in this task are ambiguous
ones such as "Go to the room with the bed" rather than direct
actions such as "Go right". In this study, we use VDNs to capture
the images of the virtual robot and the ambiguous speech of theuser to guide the robot to the location. We aim to estimate andguide the user to the location based on more ambiguous andeveryday speech such as "I'm hungry" than the dialogue
examples assumed in the VDN.

2.2 Research on Common Sense
Previous researches on common sense knowledge include
COMET-ATOMIC [2], a commonsense knowledge graph with1.33 million English entities and tuples of everyday reasoning
knowledge about events.

KGR4XAI'21, December, 2021, Online

A. Schalkwijk et al.

COMET-ATOMIC enables us to reason about everyday
events and objects, as well as human behavior and mental
states in relation to certain events, based on 23 kinds ofcommon-sense relations. Using this knowledge graph, we havealso created a language model that performs inference basedon two inputs: entity pointing at an event or a thing, andpredicate denoting a relation.
In this research, we use this learned language model to infer
the location of a guide from the user's speech. In this study, we
use three types of common-sense relations: "events thathumans cause in response to certain events," "actions thathumans need to take in response to certain events," and "places
where certain objects may be found".

3 Proposed System
In this section, we describe the structure of the proposedsystem, the construction procedure of the home environment
ontology, and the dialogue based on common sense reasoning.
In the proposed system, a user inputs a sentence to thesystem, and a virtual robot responds and guides the user
through the virtual home environment based on the dialogue
rules. We assume that the user and the virtual robot arewalking together in the virtual home environment, and thatthey are looking at the same scenery (images of the virtualenvironment sensed by the virtual robot's camera).

3.1 System configurationFigure 1 shows the system configuration for this research. First,
the conversation between the user and the system is conductedin text format through the interface. When a user inputs anutterance, the intention and entity of the utterance are
extracted through the natural language understanding modulein the dialogue system. The dialogue management module thendetermines the system's response based on the intentions,entities, and dialogue rules. When a virtual robot is to guide theuser based on the dialogue rules, an action script is specified tomove the robot, and the action is executed on the Embodied AI
simulator based on the script. In case the user's speech is
ambiguous and it is difficult to specify the location of theguidance, we use a reasoning module that utilizes the homeenvironment ontology and common-sense reasoning to infer
the location of the guidance from the speech.

3.2 Interface
In this study, we use Telegram 1 as an interface, which is an
instant messaging application that allows users to talk by text
input. We chose Telegram because it is easy to install andwidely used, and because it can be linked with Rasa [5], thedialogue framework used in this study.

3.3 Dialogue System
To build this dialogue system, we use Rasa [5], an open-source
dialogue framework designed to develop task-oriented
dialogue systems.
As shown in Fig. 2, Rasa prepares a data set of example
dialogues corresponding to certain intentions created by theuser, and by learning from the data set attached with prediction
labels, it is able to extract intentions and entities from the user'sinput. Then, dialogue is performed based on dialogue rules. In
the dialogue rules, the system can determine the output
content of the system based on the intentions extracted fromthe user's input. Furthermore, specific programs written inPython can be executed based on the dialogue rules.
Next, we discuss our reasons for selecting Rasa. We
surveyed the latest papers on dialogue system frameworks [7,9] and compared Rasa with other frameworks. As a result, wefound that Rasa is superior in the following points:

Figure 1: System configuration diagram1

https://telegram.org

Development of an interactive navigation system based onhousehold ontology and commonsense reasoning

KGR4XAI'21, December, 2021, Online

Figure 2: Example of system input/output- It can extract intentions and entities from user input
sentences.
- Ability to configure dialogue rules and slots- Capable of executing programs created by the user based on
dialogue rules- Can be integrated with existing interfaces- The development scale is large, and the framework is well
documented and easy to use.
For these reasons, we decided to use Rasa for thedevelopment of the dialogue system in this study.

3.4 Embodied AI Simulator
We use VirtualHome [6] for the Embodied AI simulator, which
is a simulator that aims to simulate activities in a virtual home.
The environment of VirtualHome is represented as aknowledge graph, which contains environment data such as
room and object identification numbers, location coordinates,and object states, in the form of a dictionary with data names
as keys and corresponding values. In this research, we use thisenvironmental knowledge to construct a home environment
ontology.
In VirtualHome, specific tasks can be performed using
action scripts. As shown in Fig. 3, an action script consists of theactions of the virtual robot, the rooms and objects to be acted
upon, and the identification numbers of the rooms and objects.
In this research, this action script is used to guide the robot.
We compared VirtualHome with other Embodied AI
simulators, referring to a survey paper on the latest Embodied
AI simulators [8]. As a result, we found that VirtualHome hasrich environment data about objects in the environment. As aresult, we decided to use VirtualHome because it is superior in
the following three aspects: it has rich environment data about

Figure 3: Example of action script

objects in the environment, it is easy for virtual robots to moveto specific rooms and objects, and it provides a virtual homeenvironment.

3.5 Home environment ontology
There is prior work on ontologies for behavior in
VirtualHome[13].In this study, we created a new ontology
because we need information such as properties and Japaneselabels for objects and rooms that exist in VirtualHome.
Figure 4 shows the construction procedure of the homeenvironment ontology. In Step 1 of Figure 4, we extract thenames of rooms and objects from all environment data in theVirtualHome knowledge graph. In Step 2, we add the extracted
data as classes of the ontology using DODDLE-OWL[3] andWordNet[4]. In Step 3, Japanese labels are added to the addedclasses in order to perform Japanese dialogues. Next, in Step 4,
all environment data is generated as an instance of the addedclass. In Step 5, we define properties to store environment datasuch as location coordinates, identification number, and status
in the created instance. In addition, to facilitate guidance, weadded a property to indicate in which room an object exists. In
the end, the number of classes, properties, and instances of thehome environment ontology created was 239, 6, and 356,respectively. Figure 5 shows part of the class hierarchy of thecreated ontology, and Table 1 shows the properties and theirdefinition and value ranges. This ontology is written inRDF/XML format.

3.6 Inference of Speech Content
Figure 6 shows the flow from user speech to location guidance
based on the home environment ontology and common-sense
reasoning.When a user inputs an ambiguous utterance such as"I am Hungry" in Japanese. The system translates it into English
as "I am hungry" in order to perform inference using COMETATOMIC[2]. Next, the translated English sentence is input into
COMET-ATOMIC, and the system uses the three common sense

KGR4XAI'21, December, 2021, Online

A. Schalkwijk et al.

Figure 4: Home environment knowledge construction procedure

Table1: Properties and their domains and ranges

Figure 5: Class hierarchy

Development of an interactive navigation system based onhousehold ontology and commonsense reasoning
relations of "events that humans cause in response to certain
events," "actions that humans need to take in response tocertain events," and "places where certain objects may befound" to infer where the user wants to go and what actions totake. As a result, output results such as "go to kitchen" and "eatfood" are obtained. The system then extracts nouns from theKGR4XAI'21, December, 2021, Online
output and obtains the names of places the user wants to go andthings the user wants, such as "kitchen" and "food". The
obtained names are checked against the classes and subclasses
in the home environment ontology. The system obtains theproperty values and Japanese labels for the location of the roomfrom the instances of the matched class, and presents the user

Figure 6: The flow of location guidance from user's speech based on

Figure 7: Example of dialogue of the proposed system

KGR4XAI'21, December, 2021, Online

A. Schalkwijk et al.

Figure 8: Result of simulation
with a list of candidates, such as "There are cupcakes in thekitchen. The user then interacts with the system to narrow
down the location to be guided. When the location is decided,
the system obtains the identification number from the instanceand guides the user using an action script.
In this way, the system presents several candidates for
guidance based on the inference results of the user's
ambiguous speech, and the user selects one from them.

4 Execution Example of the Proposed System
Figure 7 shows an example of the proposed system using
Telegram. When a user says "I'm hungry" in a dialogue, Rasa's
natural language understanding component extracts the user's
intention to be hungry from the input sentence. Based on theextracted intention and the dialogue rules, the inference
module is used to infer the location of the guide, and theprogram is executed to identify the location throughinteraction with the user. The system outputs the information
about what is in which room as a candidate for guidance. Next,
if the user specifies multiple objects in different rooms, and thesystem is unable to identify one, it prompts the user to choose
which room to be guided to. Finally, if the user is able to identify
the room and the location of the object to be guided, the system
will guide the user to the desired location through a simulation
using VirtualHome, as shown in Figure 8.

5 Policy of the Evaluation ExperimentAccording to a survey paper on task-oriented dialogue systems[7], the following three methods are used for evaluation
experiments.
1. automatic evaluation
Evaluation methods based on automatic evaluation criteria
for components in a system
2. simulation-based evaluation
Evaluation by building a user simulator using domain
knowledge and providing human-like conversations.
3. human evaluation
Evaluation by having a participant complete a specific task ofinteracting with the system and evaluating the experience.
In our dialogue system, since the guidance is based on thedialogue with the system on the Embodied AI simulator, humanevaluation is necessary to determine whether the guidance issuccessful or not. The main evaluation items are the task
success rate, which indicates whether the guidance issuccessful or not, the irrelevant turn rate, which indicates
whether there are unnecessary dialogues in the process until
the success of the task, and the user's satisfaction with thesystem.

Development of an interactive navigation system based onhousehold ontology and commonsense reasoning

6 Conclusion
In this paper, we described the construction andimplementation of an interactive dialog navigation system
based on home environment ontology and common-sensereasoning.
The functionality of this system is limited to navigation anddoes not support other actions provided by VirtualHome. Also,
currently, we do not support questions such as How and Why
for user input.
In the future, we expect to apply this research to thedevelopment of household robots and real estate property
guidance robots by reflecting the virtual robots learned in thevirtual space in the real world. We plan to add dialogue rules
and natural language understanding data sets to the dialoguesystem to enrich the conversation patterns, and to conduct theevaluation experiments described in Section 5. In addition,
after conducting evaluation experiments, we would like toclarify the technical issues and solutions of this system.
We plan to use the Corpus of Everyday JapaneseConversation (CEJC) [10] to expand the dataset of dialogue
rules and natural language understanding. the CEJC is a corpus
of natural everyday conversations in various situations. In thisstudy, we are considering extending the dataset with a corpusof daily conversations at home in order to conduct dialogues
about the home. In addition, although the ontology we havecreated now provides information about objects that exist in
the environment, we will consider using an ontology about
activities in the future, as in the research of [12].

REFERENCES[1] J. Thomason ,et al . Vision-and-Dialog Navigation. CoRL2019.
[2] D. Jena Hwangy , et al. (COMET-)ATOMIC2020: On Symbolic and Neural
Commonsense Knowledge Graphs. AAAI2020.
[3] T. Morita, N. Fukuta,N. Izumi, T. Yamaguchi: DODDLE-OWL: Interactive
Domain Ontology Development with Open Source Software in Java, IEICE
Transactions on Information and Systems, Special Section on KnowledgeBased Software Engineering, Vol. E91-D No. 4 pp. 945-958 DOI: 10.
1093/ietisy/e91-d. 4. 945. 2018.
[4] C. Fellbaum. WordNet and wordnets. In: K. Brown et al. Encyclopedia ofLanguage and Linguistics, Second Edition, Oxford: Elsevier, 665-670. 2005.
[5] T. Bocklisch , et al, A. Nichol. Rasa: Open Source Language Understanding andDialogue Management. NIPS 2017 Conversational AI workshop. 2017.
[6] X. Puig, et al .“VirtualHome: Simulating household activities via pro-grams,”
Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.
8494–8502. 2018
[7] Z. Zhang, et al. Recent Advances and Challenges in Task-oriented Dialog
Systems. Sci. China Technol. Sci. 63, 2011–2027 (2020).
[8] J. Duan, et al. A SURVEY OF EMBODIED AI: FROM SIMULATORS TO RESEARCHTASKS. CVIU2021[9] M. Burtsev, et al. DeepPavlov: Open-Source Library for Dialogue Systems.
Proc. the 56th Annual Meeting of the Association for Computational
Linguistics-System Demonstrations, pp. 122–127, 2018.
[10] H. Koiso, Y. Iseki, Y. Usuda, W. Kashiwano, Y. Kawabata, Y. Tanaka, Y. Den andK. Nishikawa. Construction of "Japanese Corpus of Daily Conversation". In
Proceedings of the 23rd Annual Conference of the Association for Natural
Language Processing (NLP2017). (in Japanese)[11] T. Pejsa, J. Kantor, H. Benko, E. Ofek and A. Wilson: Room2room: Enabling
life-size telepresence in a projected augmented reality environment. Proc. of19th ACM-CSCW, pp. 1716-1725, 2016.
[12] S. Egami, S. Nishimura. and K. Fukuda.: VirtualHome2KG: Constructing andAugmenting Knowledge Graphs of Daily Activities Using Virtual Space. In:

KGR4XAI'21, December, 2021, Online
International Semantic Web Conference (ISWC) 2021: Posters, Demos, andIndustry Tracks. Vol.2980, CEUR-WS (2021)[13] A. Vassiliades., N. Bassiliades. F. Gouidis and T. Patkos.: A knowledge
retrieval framework for household objects and actions with external
knowledge. In: International Coanference on Semantic Systems. pp. 36–52.
Springer, Cham (2020)