言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Generating Natural-Language Navigation Instructions
from Panoramic Images
Erick Mendieta1,2 Naoaki Okazaki 1,2 Hiroya Takamura1,2
1
Tokyo Institute of Technology 2 AIST
{erick.mendieta,naoaki.okazaki} at nlp.c.titech.ac.jp
takamura.hiroya at aist.go.jp
1Introduction

The goal of the Vision-and-Language Navigation
task (VLN) [1] is to train an agent to move througha lifelike environment using visual cues and a naturallanguage instruction. The agent is required to understand the provided instruction and decide whichaction to take in order to get as close as possible tothe target location. An example instruction can befound in Figure 1. The agent must be able to follow
the commanded instruction even in previously unexplored environments.
Following navigation instructions is rather trivial
for humans but hard for an agent [1, 4]. Humans
tend to subjectively select reference points in the environment to describe a path. Therefore, we end up
with a diverse set of instructions from the same sequence of images in the path. This can explain why it
is difficult for an artificial intelligent agent to replicate
humans in this task.
Previous approaches to this task [4] include aspeaker-follower model. The speaker part of themodel is a method for generating navigation instructions from panoramic images and actions in a path.
The speaker model alone is useful because it canbe used to interesting and important real-life applications such as autonomous driving navigation andvirtual-reality navigation.
In this work, we focus on the navigation instruction
generation problem. We extend the speaker model bytaking into account the human behavior of selecting
diverse reference points. In addition, we consider thefact that each view angle in a panoramic image corresponds to a time step on the simulator, representing
actions Left, Right, Up, Down, Forward, and Stop.

Instruction 1: Turn left
and to towards thekitchen, turn left at thekitchen and go until you
are right before theyellow couch by thepicture on the right.
Instruction 2: Leave
the dining room throughthe closest door andenter the kitchen. Turnleft and walk straight
down the room past therefrigerator, dining area,and sitting area. Stop infront of the fireplace.
Instruction
3:Walkforward into the kitchen
and take a left. Walkthrough the kitchen. Stop
next to the fireplace inthe living room.

Figure 1: An example path for the VLN task. The
arrows beside the images represent actions in the environment (e.g., left, right, up, down, forward, stop).
Wang et al [10] noted this fact, and suggested that
the VLN task has a sequential-decision making nature. The speaker model, however, uses an attention
mechanism over all the viewpoints without considering the sequential nature of the task. To resolve thisproblem we create a panorama encoder which relies
on a series of LSTM layers with attention to capture the sequential nature of panoramic images. Theencoder is then used together with the Transformer
encoder-decoder model [9] to generate navigation instructions.

2

Related Work

Four formal research papers have been presented
so far to undertake this problem. The first work [1]introduced the VLN task, built the Room-2-Room(R2R) dataset, and presented a sequence-to-sequencebaseline. Their focus was to improve the accuracy oftask performance of the agent. This differs from ourmain goal, which is to produce natural-language nav-― 739 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

igation instructions from panoramic images. Wanget al [10] presented a recurrent policy network withlook-ahead modules that look at the adjacent view
angles to take advantage of the visual diversity of thepanoramic images. However, similarly to Anderson
et al. [1], they focused only on increasing the performance of the agent.
The third paper [4] proposed the Speaker-Followerapproach, in which the speaker generates navigation
instructions from panoramic images. The follower
takes a decision by comparing the similarity scores between the generated instructions by the speaker andthe golden instruction. In this paper, we use this work
as a basis.
The last work [8] uses the attention mechanism intransformer [9] over the text to track the progressof the agent in the VLN task. They also use visual
attention over the images to extract features, and givethe result to a progress monitoring module. They
do not generate instructions from panoramic images,
which is the focus of our research.

3Methodology

3.1 Panoramic action space
Similarly to [4], we define a panoramic action space.
A path P contains n viewpoints v0 , v1 , ..., vn . A viewpoint vi is formed by a discretized panoramic image,which has been divided into 36 view angles denotedby k = 36. Concretely, there are 3 elevations (top,middle, bottom) and 12 headings per elevation. In
this work, we represent the view angles as elevation
vectors T = (t0 , t1 , ..., t11 ), M = (m0 , m1 , ..., m11 ),B = (b0 , b1 , ..., b11 ) for top, middle and bottom, respectively. With this notation, we represent a viewpoint vi = (T, M, B). A vertical segment vsi is defined by the top, middle and bottom view angles at
position i; for example vs0 = (t0 , m0 , b0 ). In total,
we have 12 vertical segments. The action space contains 6 different possibilities: Left, Right, Up, Down,Forward, and Stop. We set the dimension d of thehidden layer to 256.
3.2 Panoramic encoding modelThe panoramic encoder consists of four left-toright LSTM layers L1, L2, L3, L4. The first 3 layers L1, L2, L3 process the elevation vectors T, M, Bof the time steps independently of elevations, i.e.,(t0 , t1 , ..., t11 ) in L1, (m0 , m1 , ..., m11 ) in L2, and(b0 , b1 , ..., b11 ) in L3. The intention is to capture thesequential dependencies between each time step andits neighbors.
Then, the unrolled outputs of L1, L2, L3 are
A1, A2, A3, respectively:
A1 = (a10 , a11 , ..., a111 ),A2 = (a20 , a21 , ..., a211 ),A3 = (a30 , a31 , ..., a311 ).
The outputs from the 3 layers are concatenated in order to form a vector C = (A1; A2; A3). We then apply
a Masked Multi-head attention mechanism (MMAH)
with eight attention heads h = 8 as described inTransformer [9]. Here the query, key and value vectors Q, K, V from the MMAH are all set to C. For
the mask we have a vector IM = (im0 , ...imk ), with
imi = 1 if we can access the next viewpoint fromthe i-th image, and imi = 0 otherwise. Each imvalue is determined by the output of the simulation environment. The output of MMAH is given as(a1′0 ; a1′1 ; ...; a1′11 ; a2′0 ; a2′1 ; ...; a2′11 ; a3′0 ; a3′1 ; ...; a3′11 ).
For the last layer L4, we reshape the output ofMMAH to form vertical segment headings vs in thesame way as described in Section 3.1. We obtain vector VS = (vs0 , vs1 , ..., vs11 ) with vsi = (a1′i , a2′i , a3′i ),which is fed to L4 layer to obtain the vertical sequential relationships in the panoramic image with respect
to its horizontal counterparts. A second Multi-headattention layer MMAV is used to attend the verticalrelationships.
The next step uses a feed forward layer with size(n ∗ 12 ∗ d, tsl ∗ d) to reshape the output of MMAV tothe required token sequence length tsl = 50 + 1. We
add one token for the “<EOS>” token.
Finally, the output of the feed-forward layer is fed
to the transformer model as described in the previouswork [9]. We obtain the predicted tokens by performing the greedy decoding over the output.

4Experiments

We first evaluate our model in terms of its ability togenerate navigation instructions from panoramic images. We present an example from a randomly generated path of an unseen building. We then use BLEU

― 740 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Feed Forward
vs0

L4 LSTM

Cvs1

vs0

vs1

vs2

vs2

vs3

vs4vs3

vs4

vs5

vs5

vs6

vs6

vs7

vs7

vs8

vs8

vs9

vs9

vs10

vs10

a10

a11

a12a13a14a15a16a17a18a19a110

a111a21a22a23a24

a25a26a27a28a29a210a211

a30

a31a32a33a34a35a36a37a38a39a310

a311

L1 LSTM

t0

t1

t2

t3

t4

t5

t6

t7t8t9

t10

t11

A1

L2 LSTM

m0m1

m2

m3

m4

m5

m6

m7

m8

m9

m10

m11

A2

L3 LSTM

b0

b1

b2

b3b4b5b6b7b8b9b10

b11

A3

MMA1Topb1

Speaker

28.3

27.5

Ours

28.5

27.0

matic inference in the VLN task.
Implementation Details: Following the previousresearch, we use the pretrained visual feature vectors
provided by Anderson et al. [1]. These features originate from the final layer of a ResNet-152 [5] trained onImageNet [3]. We use the negative log likelihood lossand Adam optimizer with default parameters. We
use a batch size of 16, hidden size of 256. We train
for 30,000 iterations. We use the greedy strategy fordecoding.

Middleb0

Val_unseen

Table 1: BLEU score (%) of the baseline and thepresent work

A4

a20

BottomVal_seen

MMA2vs11vs11

Modelb2

Figure 2: Panoramic encoding model.
score to compare our results with a baseline. We also
evaluate the use of visual diversity of the panoramic
image; we select a path in which multiple view angles
can reach the next viewpoint. We compare generated
instruction by the proposed method, the baseline, andhumans (golden instructions).
R2RDataset: The Room-to-Room dataset [1] consists of 10,800 viewpoints constructed from 194,400
RGB-D images of 90 buildings from the matterport
dataset [2]. In addition, selected 7,189 sample paths(5-7 viewpoints each) are associated with three navigation instructions (21,567 instructions in total) collected from Amazon Mechanical Turk (AMT). The
average instruction length is 29 words. The vocabulary size is around 3.1k words (1.2k with 5 or morementions). We use the dataset splits reported in [1]:training (14,025 instructions), validation seen (1,020),
validation unseen (2,349) and test (4,173).
Evaluation Metrics: We report results with thestandard BLEU score metric [6] in the validation seen
and validation unseen splits. For the qualitative evaluation, we show a comparison of the generated sentences between the baseline, our model, and reference(golden).
Baseline: As a baseline for our work, we selected
the speaker model presented by Fried et al [4], whose
purpose was to perform data augmentation and prag-5

Results and Analysis

We first report the comparison of the BLEU score.
In terms of the BLEU score, our method performs
better than the baseline in the validation seen split
by a small margin. This small margin suggests that
our method is comparable with the previous work. In
the Validation unseen, however, our method performs
slightly worse. We think that the reason for the small
loss margin in the validation unseen is because we
did not use pretrained GloVe embeddings [7] for thetarget words as the speaker model did. To further
analyze the results, we perform a comparison between
the good and bad examples predicted by the baselineand our work.
Similarly to the baseline, our method can generate
instructions from paths in unknown buildings. The
example with path id 1001358 comes from the data
augmentation list provided by Fried et al. [4]. Figure 3 shows the instruction generated by our method.
While these two sentences are similar to each other,ours tends to generate tokens that more closely refer to the objects seen in the picture. For example,
in the same figure, our algorithm refers to the door
as glass door, which is appropriate. Instead the previous research use the most generic term ”doorway”.
We observe the same at the end of the sentence withthe use of the phrase ”Wait on the porch”.

― 741 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Previous work: walk down
the hallway and turn right .
walk down the hall and stopin the doorway .

Our work: walk up the stairs. walk past the pillars on theleft . wait at the edge of therug .

Our work: turn around andgo out the door . go out theglass door and go outside .
wait on the porch .

Previous work: go up thestairs and go up the stairs .
go up the stairs and stop at
the top of the stairs .
Golden instruction: Ascendstairs, then walk toward thekitchen area, and wait.

Figure 3: Example 1001358 speaker augmentation
split, Both algorithm are able to generate instructions
for unseen case. However, ours better identify the objects by their names

Golden: Exit the kitchenheading
towards
thesmall dining nook andturnright.
Continue
forward and take theentrance ahead and tothe right. Wait at thecorner next to the end ofboth couches.

Figure 5: Example 5412-0 Validation unseen. We
show the most difficult examples for which ourmethod has a slightly better advantage by making
use of the sequential nature of the panoramic images.

6Conclusion

We presented a method for generating navigation
instructions from panoramic images that takes advantage of the sequential nature of the VLN task.

work: turnaroundandwalkthrough the kitchen .
turn right and walk past
the dining table . stopin front of the dining
table .
Previous

Our work: walk into thekitchen and turn right .
walk past the table andinto the living room .
stop in front of thefireplace .

Acknowledgment This work is supported byRWBC-OIL.

Figure 4: Example 2068-0 validation unseen. Our
method predicts “fireplace” found in the images, although it is not mentioned in the golden navigation
instruction.

References
[1]

[2]

Our method also seems to recognize objects that are
not present in the golden instruction but are shownin the image. Figure 4 provides such an example, inwhich the word “fireplace” appears in our prediction,
but not in either the golden instruction nor the generated instruction generated by the previous work.
More confusing examples are the ones that contains
a repetition of actions. For example, when there is astaircase with multiple stair flights to climb, the algorithm in the previous work tends to repeat the same
instruction over and over again. The same thing happens when the environment contains multiple doors
in the same place. Our method, however, benefits
from the sequential nature of the task and the useof panoramic information to better represent the scenario. See Figure 5.

[3]
[4]
[5]
[6]
[7]

[8]

[9]
[10]

― 742 ―

Peter Anderson et al. “Vision-and-Language Navigation:
Interpreting visually-grounded navigation instructions inreal environments”. In: CVPR. 2018.
Angel Chang et al. “Matterport3D: Learning from RGBD Data in Indoor Environments”. In: 3DV . 2017.
Jia Deng et al. “ImageNet: A Large-Scale Hierarchical
Image Database”. In: CVPR. 2009.
Fried et al. “Speaker-Follower Models for Vision-andLanguage Navigation”. In: NeurIPS. 2018.
Kaiming He et al. “Deep Residual Learning for Image
Recognition”. In: CVPR. 2015.
Kishore Papineni et al. “BLEU: a Method for Automatic
Evaluation of Machine Translation”. In: ACL. 2002.
Jeffrey Pennington, Richard Socher, and Christopher D.
Manning. “Glove: Global vectors for word representation”. In: EMNLP. 2014.
anonymous - under review. “Self-Monitoring Navigation
Agent via Auxiliary Progress Estimation”. In: ICLR.
2019.
Ashish Vaswani et al. “Attention Is All You Need”. In:NeurIPS. 2017.
Xin Wang et al. “Look Before You Leap: Bridging
Model-Free and Model-Based Reinforcement Learning
for Planned-Ahead Vision-and-Language Navigation”.
In: ECCV . 2018.

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.