DialMAT: Dialogue-Enabled Transformer withMoment-Based Adversarial Training
Kanta Kaneda⋆ , Ryosuke Korekata⋆ , Yuiga Wada⋆ , Shunya Nagashima⋆ , Motonari Kambara,
Yui Iioka, Haruka Matsuo, Yuto Imai, Takayuki Nishimura, and Komei Sugiura
Keio University{k.kaneda, rkorekata, yuiga, ng sh, motonari.k714, kmngrd1805,haruka.matsuo-25, ytim8812, t-nishimura, komei.sugiura}@keio.jp

1. Introduction
In this paper, we focus on the task of embodied instruction following in a setting where an agent can ask questions
to the human and utilize the information provided in the response to enhance its ability to complete the task effectively.
The main contributions of this paper are as follows:• We introduce Moment-based Adversarial Training(MAT) [4] to incorporate adversarial perturbations into
the latent space of language, image, and action.
• We introduce a crossmodal parallel feature extraction
mechanism to both language and image using foundation models [3, 6].

2. Problem Statement
This study focuses on the DialFRED task [1], which
involves embodied instruction following. In this task, anagent has the ability to actively ask questions to the humanuser and utilize the information provided in the response
to enhance its effectiveness in completing the task. DialFRED is based on the standard benchmark ALFRED [7]* Equal contribution

CLIP
To move twobooks to thebed.

MAT
DeBERTa

CLIP
MAT
ResNetFC

This paper focuses on the DialFRED task, which is thetask of embodied instruction following in a setting wherean agent can actively ask questions about the task. To address this task, we propose DialMAT. DialMAT introducesMoment-based Adversarial Training, which incorporates
adversarial perturbations into the latent space of language,
image, and action. Additionally, it introduces a crossmodal
parallel feature extraction mechanism that applies foundation models to both language and image. We evaluatedour model using a dataset constructed from the DialFRED
dataset and demonstrated superior performance comparedto the baseline method in terms of success rate and pathweighted success rate. The model secured the top positionin the DialFRED Challenge, which took place at the CVPR
2023 Embodied AI workshop.

Positional
Encoding

Instruction & Answer

Transformer

Abstract

previous action

Pick up Book

MAT

Figure 1. DialMAT consists of two main modules: Questionerand MAPer. Our method employs MAT to incorporate adversarial
perturbations into the latent space of language, image, and action.

for Vision-and-Language Navigation tasks involving object
manipulation. In the DialFRED setting, the robot can ask
questions about the position of the object, its description,and the direction in which it should move.
The input and output for this task are defined as follows:• Input: Instructions for each subgoal, answers, andRGB images for each timestep.
• Output: Action taken at each timestep.

3. Method
We propose DialMAT* , an extension of the Episodic
Transformer [5]. The proposed method consists of two main
modules: Questioner and Moment-based Adversarial Performer (MAPer). Fig. 1 shows the overview of DialMAT.
The input xt for time t in our model is defined as xt =(k) (k)(D(k) , l(k) , vt , ât−1 ), D(k) = {(qi , si ) | i = 0. . . N },(k)where l , vt and ât−1 denote the instruction for the kth
subgoal, perspective RGB image at time t, and previous action at time t − 1, respectively. Here, â is composed of apair of action types and manipulated objects. Note that depending on the type of action, there may or may not be a
* https://github.com/keio-smilab23/DialMAT

manipulated object involved. In addition, D(k) refers to theset of question-answer statements in the kth subgoal, which(k)is composed of pairs of question statements qi and corre(k)sponding response statements si .
First, the Questioner determines which question needs tobe asked at time t. This module has the same structure as theQuestioner proposed in [1], and is composed of an LSTM
with an attention mechanism. The input in Questioner isl(k) , and LSTM encoder and decoder perform multi-levelclassification. That is, at time t, it determines which of theLocation, Appearance, and Direction questions to ask, andobtains the response s(k) for each question.
Second, the MAPer takes xt as input and outputs therobot’s behavior at time t. First, it computes the respective
embedded representations hctxt and hdeb from l(k) using
CLIP [6] and DeBERTa v3 [3]. Then, following the MATapproach, we obtain the features htxt by adding a learnable
perturbation δtxt as htxt = [hctxt ; hdeb ] + δtxt . Note that
δtxt is updated based on the following steps. First, we compute the gradient of the loss function E with respect to theperturbation ∇δ E. Next, we introduce two types of moving
averages using ∇δ E as follows:
mt = ρ1 mt−1 + (1 − ρ1 )∇δ E(δt ),vt = ρ2 vt−1 + (1 − ρ2 )(∇δ E(δt ))2
where t denotes the current perturbation update step andρ1 , ρ2 denote the smoothing coefficients for each moving
average. Finally, using the above mi , vi , the update widthof the perturbation ∆δi is calculated as follows:vtm̂t
mt, v̂t =, ∆δt = η √,m̂t =1 − (ρ1 )t
1 − (ρ2 )tv̂t + ε∆δt),δt+1 = Π∥δ∥≤ϵ (δt +∥∆δt ∥F
where η, ϵ, Π∥·∥≤ϵ and ∥ · ∥F denote the learning rate of theMAT, minute value that prevents zero division, the projection onto the ϵ-sphere and the Frobenius norm, respectively.
Next, we compute the respective embedded representations
hcimg and hres from CLIP and ResNet [2]. Then, we apply
MAT to obtain the feature himg by adding the perturbation
δimg as himg = [hcimg ; hres ] + δimg . Similarly, for s(k) andat−1 , MAT is applied to obtain the respective latent representations hans and hact . The following embedded representation h1 is then input to the N -layer Transformer toobtain h(N ) :h1 = [htxt ; hans ; himg ; hact ] + Epos ,hi = Transformer(hi−1 ),where Epos refers to positional encoding and embeds the positional information of each token in[htxt ; hans ; himg ; hact ]. Finally, we obtain the predicted action ât by applying the fully connected layer toh(N ) . We define the loss function as the cross-entropy
between ât and expert action at .

Table 1. Quantitative comparison. The best scores are in bold.

MethodBaseline [1]Ours (w/o MAT [4])Ours (w/ CLIP text encoder [6])Ours (MAT for action)
Ours (DialMAT)

Pseudo TestSR↑ PWSR↑
0.31
0.19
0.34
0.20
0.35
0.22
0.36
0.21
0.39
0.23

Test
SR↑
0.14

Figure 2. A successful subgoal prediction. In this case, the instruction was “Move to the floor lamp, power on the floor lamp.”

4. Experimental Setup
In this study, we employ the same experimental settings
as DialFRED [1]. In this study, we divided the DialFRED
dataset as described in [1]. As the test set of DialFRED is
not publicly accessible, we further divided the valid unseen
set to pseudo valid and pseudo test set, comprising 685 and678 tasks, respectively. We used the training set to train theMAPer and the valid seen set for reinforcement learning ofthe Questioner.

5. Experimental Results
Table 1 shows the quantitative results of the baseline andthe proposed methods. We evaluated the models by successrate (SR) and path weighted success rate (PWSR). Table 1
shows that the SR on the pseudo test set for the baseline
and the proposed methods were 0.31 and 0.39, respectively.
Therefore, the proposed method outperformed the baseline
by 0.08 points in SR. Similarly, the proposed method also
outperformed the baseline method in PWSR.
Fig. 2 shows the qualitative results. In this sample, theinstruction was “Move to the floor lamp, power on the floorlamp.” In this case, the robot was required to move to thefloor lamp and then turn it on. The proposed method wasable to appropriately navigate to the floor lamp and successfully turn it on.

6. Conclusions
In summary, the contributions of this work are twofold:• We introduced MAT to incorporate adversarial perturbations into the latent spaces.
• We introduced crossmodal parallel feature extraction
mechanisms to both language and image using foundation models.
Acknowledgements This work was partially supported by
JSPS KAKENHI Grant Number JP23H03478, JST Moonshot, and NEDO.

References[1] Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, et al.
DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following. IEEE RA-L, 7(4):10049–10056, 2022. 1, 2
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep Residual Learning for Image Recognition. In CVPR,pages 770–778, 2016. 2[3] Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using ELECTRA-Style PreTraining with Gradient-Disentangled Embedding Sharing. InICLR, 2023. 1, 2
[4] Shintaro Ishikawa and Komei Sugiura. Moment-based Adversarial Training for Embodied Language Comprehension. InIEEE ICPR, pages 4139–4145, 2022. 1, 2
[5] Alexander Pashevich, Cordelia Schmid, and Chen Sun.
Episodic Transformer for Vision-and-Language Navigation.
In ICCV, pages 15942–15952, 2021. 1[6] Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning
Transferable Visual Models From Natural Language Supervision. In ICML, pages 8748–8763, 2021. 1, 2
[7] Mohit Shridhar, Jesse Thomason, Daniel Gordon, et al. ALFRED: A Benchmark for Interpreting Grounded Instructions
for Everyday Tasks. In CVPR, pages 10740–10749, 2020. 1