Learning Rewards and Skills to Follow Commands with a Data Efficient Visual-Audio Representation
Authors: Peixin Chang
Shuijing Liu
Tianchen Ji
Neeloy Chakraborty
D Livingston McPherson
Katherine Rose Driggs-Campbell
Venue: Submitted to ICLR 2023 (2023)
Keywords: Robotics
Representation Learning
Reinforcement Learning

Abstract: Based on the recent advancements in representation learning, we propose a novel framework for command-following robots with raw sensor inputs. Previous RL-based methods are either difficult to continuously improve after the deployment or require a large number of new labels during the fine-tuning. Motivated by (self-)supervised contrastive learning literature, we propose a novel representation, named VAR++, that generates an intrinsic reward function for command-following robot tasks by associating images with sound commands. After the robot is deployed in a new domain, the representation can be updated intuitively and data-efficiently by non-expert, and the robots are able to fulfill sound commands without any hand-crafted reward functions. We demonstrate our approach to various sound types and robotic tasks, including navigation and manipulation with raw sensor inputs. In the simulated experiments, we show that our system can continually self-improve in previously unseen scenarios given fewer new labeled data, yet achieves better performance, compared with previous methods.



Under review as a conference paper at ICLR 2023
LEARNING REWARDS AND SKILLS TO FOLLOW COMMANDS WITH A DATA
EFFICIENT VISUAL-AUDIO REPRESENTATION
Anonymous authors Paper under double-bling review
ABSTRACT
Based on the recent advancements in representation learning, we propose novel framework for command-followinn robots with raw sensor inputs. Previous RL- based methods are either difficult to continuously improve after the deployment or require a large number of new labels during the fine-tuning. Motivated by (seef--ssuerrvied contrastive learning literature, we propose novel representa- tion, named VAR++ that generates an intrinsic reward function for command- following robot tasks by associating images with sound commands. After the robot is deployed in a new domain, the representation can be updated intuitively and data-efficeently by non-experts, and the robot is able fulfill sound com- mands without any hand-crafted reward functions. We demonstrate our approach various sound types and robotic tasks, including navigation and manipulation with raw sensor inputs. In the simulated experiments, we show that our system can continually self- ceff-improee in previously unseen scenarios given fewer new labeled data, yet achieves better performance, compared with previous methods.
INTRODUCTION
When humans are told to turn on TV, they can associate what they hear with what they see even in unfamiliar environments. For robots follow commands and fulfill similar tasks, they must ground task-oriented language vision and motor skills. Command following robots is such an important application that paves the way for non-experts to intuitively communicate and collaborate with robots in daily lives.
The need for command following robots has spurred a wealth research. Learning- based language grounding agents were proposed to perform tasks according visual observations and text/speech instructions Anderson et al. 20188; Chang et al. (2020): Chaplot et al. (Hrrmmn et al.) (2017); Shridhar al. (2020); Yu al. (2018). However, these approaches fail completely solve com- mon problem in learning-
methods: performance degradation in a novel target domain (Akkaya al. 2019); y James et al.] (2019); Tobin et al. (017). One solution address the domain shift problem is domain randomization Tobin et al. (017). However, it has been shown that domain randomization alone not sufficient since the randomized simulation may not accurately reflect the target domain that the robot is later deployed in Du et al. (202T): Smith et al. (2022). Alter- natively, fine-tuning policies in the target domain can further reduce the reality gap but often cost prohibitive: professionals usually train the robots with hand-crafted, task-specific reward func- tions Haarnoja et al. (2019); /mmthh et al. (2022) and large amounts of labels, neither of which can be afforded by non-expert users after deployment.
Without enough domain expertise or abundant labeled data, how can we allow users to adapt such robots to novel domains with minimal supervision? Prior works have partially answered this ques- tion by proposing a visual-audio representation (VAR) trained with triplet loss, which associates audio commands and goal images with the same intent Chang et al. (202T). However, due to true negative pairs used in the triplet loss, the number of labels required to fine-tune the VAR is still not satisfactory, thus hindering efficient deployment in the target domain.
In this paper, we propose a novel framework that builds on belff))supervisee contrastive learning to realize more effective training and more efficient fine-tuning for rewards and skills learning. As

---

Under review as a conference paper at ICLR 2023
Put on the music?
Playy'
Reinforcement Learning
Contrastive Learning
"Biing shoes"
Rowards
VAR++
Stop the music "Pause"
Stsdee
mage
Lights on!
Turn on the lamp?
Figure 1: Illustration our pipeline. Contrastive learning used to group images and audio commands the same intent. The resulting representation VAR++ supports the downstream RL training by encoding the high-dimensional voice and image signals, and providing reward signals and states to the agent.
shown in Fig. we first learn joint representation of visual and audio signals (VAR++) whose clusters have better intra-cluster cohesion and inter-cluster separation compared to VAR Chang et al. (02T). In the second stage, we use VAR++ to compute intrinsic reward functions learn various robot skills with RL without any reward engineering. When the robot is deployed in a new domain such as a different room, the fine-tuming stage is data efficient in terms of the label usage and is natural to non-experts in terms of the human-robot interaction. For example, a user can teach robot or VAR++ by saying that "this is an apple" when the robot sees an apple. Then, RI. policies are self-im ielf-inpoveed with the updated VAR++ No hand-designed reward or negative pairs are needed as in the previous works.
We apply this learning approach different robotic tasks in diverse settings illustrated in Fig. and Fig. 3 Given a sound command, the robot must identify the commander's goal (intent), draw the correspondence between the raw visual and audio inputs, and develop policy to finish the task. The tasks are challenging because no maps, depth images, human demonstrations, or prior knowledge available, and the observation mainly comes from monocular uncalibrated RGB camera.
Our main contributions are as follows: (1) We propose a novel representation of visual-audio ob- servations for command following robots, named VAR++. To our best knowledge, it is the first work demonstrating that
contrastive loss improves robot control performance from triplet loss. (2) We propose data efficient fine-tuning method for command following robots which requires significantly fewer labels than baselines. Moreover, our fine-tuning method demonstrates that (self-) ysuperviee contrastive loss has the potential enhance user experiences, especially for non-experts. (3) We release our simulation environments and model implementations, The sim- ulations are the first open-sourced Al environments which use real speech recordings for robotic command following tasks. The code will be released after the review at www www.ggtthi. t.ub..c..oo com
2 RELATED WORKS
End-to-end language understanding. End-to-end spoken language understanding (SLU) systems extract speaker's 's intent directly from raw speech signals without translating the speech to text Kim et al. (202T): Lugosch et al. (2019): Serdyuk et al. (2018). Such an end-to-end system is able to fully exploit subtle information, such as speaker emotion, that is lost during speech to text tran- scription Kim et al. (2021): Lugosch et al. (2019). However, end-to-end SLU systems are mainly developed for virtual digital assistants and not for robotic applications.
Language grounding agents. Conventional language grounding agents consist of independent modules for transcription, language grounding, and planning MMgassouba et al. (2019;; Paul et al. 2018); Stramandinoli et al. (2016). These modular pipelines suffer from intermediate errors and do not generalize beyond their programmed domains Hermann et al. (2017): Tada et al. (2020); Vanzo et al. (016). To address these problems, end-to-end language grounding agents used to perform tasks according to text-based natural language instructions and visual observations Anderson et al.
2018): (Chplot al. 2018): Hermann et al. (2017) Shridhar et al. (2020): : Yu al. (2018). Our
2

---

Under review as a conference paper at ICLR 2023
work has two novelties from these works. First, while the above works consider text-based input, focus on commanding agents through raw audio, which leads to more natural human-robot commu- nication without additional modules. Second, training the agents requires either expert demonstra- tions, step-by-step instructions Anderson et al. 20rri;Shridhhr et al. 2200),, or carefully designed extrinsic reward function Chaplot et al. (2018): Hermann et al. (201 D Yu et al. (2018). Although some methods generalize to new command sentences and/or new scenes to some extent, they over- look the continual line-tuning after deployment [Chang et al. (2020); Yu al. 2018)) In contrast, our method requires none of the above and thus requires significanlly fewer efforts to fine-tune in a novel domain, where both perception and dynamies are different from the training scenes.
Another line of works trains the robot to fulfill commands directly from raw audio inputs with RI. Chang et al.) (2020 202Tb. However, the method in Chang et al.] (2020) requires hand-tuned reward functions and a prohibitive number of one-hot labels, which is still hard to fine-tune. Chang al. (202T) partially addresses the problem by learning visual-audio representation (VAR) with triplet loss to generate intrinsic reward function for RL. However, the quality of the VAR in Chang ct al. (202T) is suboptimal based on our quantitative evaluation. contrast, the better representation in our work leads to more ideal reward functions and better robot performance.
Representation learning for robotics. Representation learning has shown great potential in learn- ing useful embeddings for downstream robotic tasks. Deep autoencoders have been used to com- press high-dimensional observations such as images into low-dimensional latent space. The re- sulting latent vectors are then used states intrinsic rewards for Lange et al. (2012) Nair iet al. 20188;; Wang et al. (2020). At test time, however, the methods in Nair et al. (018);; Wang al. 2020 require users to provide goal images for task execution, while our method takes voice commands, which is a more natural and convenient way of human-robot communication. Addition- ally, reconstructions of the input images often make the autoencoders computationally expensive. Another line of works uses contrastive loss to learn representations for downstream tasks such as grasping and water pouring Jang et al. (2018); Nguyen et al. (200);; [Sermanet et al. (2018). Con- trastive loss avoids the reconstruction operation in autoencoders. While all of these works focus mainly on the visual or the text modality, we address the interplay between sight and sound.
3 METHODOLOGY
In this section, we describe the two-stage training pipeline and fine-tuning procedure. In training, we assume the availability of sufficiently large labeled datasets, simulators, and labels. However, in fine-tuning, speech transcriptions, one-hot labels, and reward functions, are not available.
3.1 VISUAL-AUDIO REPRESENTATION LEARNING
In the first stage, we collect visual-audio pairs from the environment. Then, we learn joint rep- resentation of images and audios, named VAR++ that associates an image with its corresponding sound command.
Data collection. Suppose there are M possible intents tasks within an environment. We collect visual-audio pairs defined as (I. S. y) from the environment, where I E mRax is the current RGB im- age from the robot's ' camera, S [ is the Mel Frequency Cepstral Coefficients (MFCC) Davis & Mermelstein (1980) of the sound command, and E {0, I,.., M) is the intent ID. We call I and two views of intent y. A visual-audio pair contains image and sound command of the same intent. For example, when an iTHOR agent sees a lit lamp. it hears the sound "Switch on the lamp" from the environment. In contrast, when the agent sees no object or is far away from all objects so that it sees many objects at once, it receives only an image and hears no sound. The image is paired with S 0/xm and y = M. We define this situation as an empty intent.
Training VAR++.
Our goal is to encode both visual and auditory signals into a joint latent space, where the embeddings from the same intents are pulled closer together than embeddings from dif- ferent intents. For example, the embedding of an image with TV turned on needs to be close to the embedding of sound command "Tum on the TV" but far away from other irrelevant com- mands such as "Tum off the light." We adopt the idea from (self- jsppevvsuee contrastive learning for visual representations and formulate the problem as metric learning. As shown in Fig. 2 the VAR++ is double-branch network with two main components. The first component is the
3

---

Under review as a conference paper at ICLR 2023
X(miss) V(xi)
LSTM
FC
VAR++
8g
pacon masic
h4
CNN
12
8N
I
Sg
M.
Encoder
Protection Heads
(8)
(c)
Figure 2: Network architectures. (a) The VAR++ is double-branch network optimized with jsupervsuee contrastive loss. (b) The latent space of the VAR++ is unit hypersphere such that the images and audios of the intent closer than those of different intent in the space. (c) The policy network for RL training. The portion blue VAR++ which frozen during RL training. We use Q denote element- addition, FC to denote fully connected layers, and [oo] to denote concatenation.
coders f R"** + Rd and f : R - Id which map an input image and a sound signal S to representation vectors h' and h respectively. In practice, any deep models for image and sound processing can be used for f and ʃs. The second component is the projection heads g' Rdi + Rd. go gds Rd b' n IR, and 6s Is tions h' and h to the space where losses are applied. We denote the vector embeddings g (h') and 9º (hS) as z' and z², respectively. We enforee the norm of z' and z to be by applying an L2- 12-normalization, such that the embeddings live on unit hypersphere as shown in Fig. 2p.
that map the representa-
We use supervised contrastive (SupCon) loss as the objective, which encourages the distance be- tween z' and z of the same intent to be closer than those of different intent Khosla et al. (2020). Suppose there are N visual-audio pairs in a batch. Let E == (1...., ...,,.N) be the index of an image or sound signal within that batch and P(k) {p K [k} Up yk be the set of
indices of all images and sounds of the same intent except for index k. Then, the SupCon loss is
exp (Zz- Zp/T)
CsupCon = -E REK P(K)
Σ log PEP(K)
(I)
ΣjER((K) exp (za 2,/T)²
where |-] is the cardinality, z(-) can be either z or S and T R is a scalar temperature parameter. The previous VAR uses visual-audio triplets of the form (I, ST) for the training, where I and St are an image and sound with the same intent, and S the sound with different intent. The loss only pulls together the embeddings of I and St and pushes away the embeddings of I and S in a triplet. This setting is less efficient because each anchor only has one positive and one negative Chang et al. 2021). In contrast, the use of SupCon loss allows the attraction and repulsion among all images and sound within a batch, which improves the performance of the representation will show in Sec. S.c. We additionally introduce binary classification loss for both the image and sound to distinguish between empty and non-empty intent. Let LECE denote the binary cross entropy loss and e denote the label of intent, which is O for empty intent and for non-empty intent. The batch loss for training the VAR++ is:
N Lucr (acr(b((h))) (hf),,e) + LBCE(b (h)),,y).
LVAR+ = Csu + 0:2 N
(2)
where and are the weights of losses. Depending on if the intent is predicted empty or not, the output v and S of VAR++ can be determined for image and sound by:
(er 20.5 z'
el b'
(3)
= 1 (25 205) S E b5=bb$(h))
where 1 is an indicator function. We call the latent space of the output as joint space. The purpose of the binary classification is set the image and sound embeddings of the empty intent the center
4

---

Under review as a conference paper at ICLR 2023
of the joint space. This centralization removes the biases caused by the location of the empty intent in the joint space, leading to better intrinsic reward generated by the VAR++.
While SupCon loss and other self- sulf-supervieed visual representation learning frameworks are origi- nally only applied to image modality Chen et al. (2020); Khosla et al. (2020), we extend the frame- work to multi-modallts setting and create new representation for command following robots.
3.2
RL WITH VISUAL--AUDIO REPRESENTATION
The second stage of our pipeline is to train an RL agent using an intrinsic reward function generated by trained VAR++ We model a robot command following task as a Markov Decision Process (MDP), defined by the tuple (x, .. P. R. 7). At each time step f, the agent receives an image I from its RGB camera, and robot states M, such as end-effector location or previous action. At = O, an additional one-time sound command Sn containing intent is given to the robot. We freeze the trainable weights of VAR++ in this stage and define the MDP state E X' as xt = [T., v v M,), where v and v the output of the VAR++ for I, and Sg. respectively. The VAR++ encodes the information in the input image and the intent in S.. Then, based on its policy x =(a [zx),, the agent takes an action de E A. In return, the agent receives a reward r. E R and transitions to the next state Fe-1 according to unknown state transition Pl az). The process continues until 1 exceeds the maximum episode length T, and the next episode starts.
Intrinsic rewards. Since v and v of the same intent are pulled together within the VAR++ by the contrastive loss, intrinsic rewards can be derived as the similarity between v and v Eq. 4 and S present two possible task-agnostic and aabot-ggnottic reward functions:
vl
(4)
=/= g V V.
(5)
where v is the embedding of the current sound signal S,, which can be triggered in the same way as S as described in Section 3.1 Intuitively, the agent using ri receives high reward when the scene it sees matches the command it hears. The agent trained using the reward rf additionally needs to match the current sound it hears with the sound command to receive high rewards. Compared to rff, the reward function rf does not depend on any real-time supervision signal such as current sound v from the environment, allowing the agent to perform self-suupervise RL training with VAR++. Although RL agents trained with Eq. 4 can already achieve decent performance, providing the current sound S, can further improve the performance Chang et al. (202T). Since S, can be difficult to obtain especially in real environments, S, is not part of the state 2, and thus the robot policy does not require S, at test time.
Policy network architecture. We show our policy network architecture used in our experiments in Fig. 2 Given the state Fı. the network outputs the value V(x;) and the policy π TTa (2). Instead of reusing the CNN in f", we add another CNN to extract the features relevant for achieving the goal. For example, the iTHOR agent needs to encode information about obstacles for collision avoidance. We use an LSTM to encode the embeddings of I, and M, for long-term decision making. Proximal Policy Optimization (PPO) was used for policy and value function learning Schulman et al. (2017).
3.3 FINE-TUNING
After the robot is deployed in new domain such as the real world, its performance often degrades due to domain shift from both perception and dynamics Du et al. (022). Our fine-tuning procedure allows non-experts to continually improve the VAR++ to reduce perception gaps and improve robot skills to reduce dynamics gaps. Since performing accurate state and reward measurements, data labeling, and instrumentation requires domain expertise and is time-consuming, we assume tuned reward functions, one-hot labels, and accurate speech transcriptions are not available from non- experts. Fortunately, our method requires none of these. To fine-tune VAR++ since we no longer have the underlying labels y for images and sounds, we replace the SupCon loss in Eq. (2) with the following self-supervised contrastive loss (SSC) Chen et al. (2020):
exp (ZA (T/(R//T) - log ΣjEK(() exp (Zx 2,/T)² AEK
Lssc =
(6)
5

---

Under review as a conference paper at ICLR 2023
where p(k) is the index of the data paired with the data of index & with the same intent. To fine-tune the robot, non-experts collect visual-audio pairs of the form (I. based on their common knowl- edge using their own voices. The robot can then self-improve its policy network with the intrinsic reward function by randomly sampling a collected sound command as the goal. See Appendix for detailed fine-tuning algorithm.
To fine-tune VAR in Chang et al. (2021), non-experts have to provide a sound command with differ- intent S for each image utilizing the SSC, leading to a more intuitive data collection experience for non-experts and better performance with fewer labeled data.
triplet loss. In contrast, VAR++ eliminates this requirement by
4 SIMULATION EXPERIMENTS
In this section, we first describe the environments (Fig. 1 and various sound datasets for the experi- ments. Then, we compare the performance and data efficiency of our pipeline with several baselines and ablation models.
4.1 ROBOTIC ENVIRONMENTS
We evaluate the performance of all the meth- ods on three different robotic platforms: Turtle- Bot, Kuka, and iTHOR. In all environments, af- ter hearing sound command, the robots must learn exploration skills and approach the corre- sponding objects. All the robots are equipped with a monocular uncalibrated RGB camera for robot perception. See Appendix C for detailed descriptions and visualizations.
ITHOR environment
TurtieBot environment
Kuka euvironment
Figure 3: Simulation environments for the experiments.
4.2 SOUND DATA
We several types of sounds from state-of-the-art datasets in training and testing. Specifi- cally, we use speech signals collected for training the end-to-end SLU from Fluent Speech Com- mands (FSC) Lugosch et al. (2019) and short speech commands from Google Speech Commands (GSC) Warden (2018). We also collect single-tone signals from NSynth Engel et al. (2017) and urban & environmental sounds from UrbanSound8K (US8K) Sallmon et al. (2014). The Wordser dataset was created from the "0." "1," "2," "3" in GSC. We also used a Mix dataset to show that the VAR++ can map multiple types of sounds to a single object or idea, by mixing speech data with environmental sound. We mix "house" with "jackhammer" and "dog" with "bark". Examples of commands in the iTHOR environment include "turn on the lights" and "pause". The iTHOR en- viromment uses the commands from FSC, while the Kuka and the TurtleBot environments uses the commands from the other sound datasets. See Appendix for more sound examples and intent we choose for the iTHOR environment.
4.3 EVALUATION OF THE REPRESENTATIONS
Evaluation metrics. The representations are evaluated by a linear layer (LL) and nearest neigh- bor (NN). For LL, follow the widely used linear evaluation protocol, where linear classifier trained using a cross-entropy loss on top of the frozen encoders, which are f and fs in our case Chen et al. (000)) Kolesnikov et al. (2019): Zhang al.) (2016). For NN, we first find the medoids of each intent Co, Cas in the joint space using the training data. The predicted label of a embedding of the test image or sound in the joint
test image or sound is arg max; v C, where space. NN measures the quality of the intrinsic reward produced by the representations.
Baselines. We compare the performance of our VAR++ with VAR which uses triplet loss for both training and fine-tuning Chang et al. (2021). For each intent, we collect the same number of visual- audio pairs (I, S, y) for VAR++ training and visual-audio triplets of the form (I,S+ S-) for VAR training, where and S are an image and sound with the same intent, is the intent ID, and S is the sound with a different intent. We kept the network architecture of both methods the same.

---

Under review as a conference paper at ICLR 2023
Quantitative results. From Table [ we ob- serve that both methods achieve high NN ac- curacy while VAR++ marginally outperforms VAR, suggesting that both methods are able to produce accurate and reliable rewards for the downstream RL tasks. As for LL, VAR++ is much better than VAR since even linear clas- sifier can achieve much higher accuracy with VAR++.
Table 1: Percentage aceuracy of VARs with near- est neighbor (NN) and linear layer (LL.).
LI
Env
Method
NN
Img
Snd
Avg
VAR
97.8
77.4
95.4
86.4
Kuka
VAR++
98.5
82.2
99.8
91.0
Qualitative results. We visualize the VARs by projecting images and sounds to the joint space,
TurtleBot
VAR
96.7
66.1
84.9
75.5
VAR++
99.1
76.5
96.8
88.1
as shown in Fig. 4 We see that the embeddings of the same concept form cluster and all clus-
VAR
96.6
51.4
91.2
73.8
iTHOR
VAR++
96.9
78.4
94.7
86.6
ters are separated from each other. Compared
to VAR, the clusters in VAR++ have better intra-cluster cohesion and inter-cluster separation, sug- gesting that the two distinct concepts are better distinguished and the same concepts better lated. During fine-tuning, although VAR++ does not have S as an explicit indication of negatives like VAR does in the input, VAR++ can still maintain relatively clear inter-cluster separation and provide reliable rewards for the self-improvement of RI. agents.
(2)
(b)
(
(d)
Figure 4: Visualizations of the VARs in the ITHOR environments with FSC. The colors indicate the ground truth intent ID of embeddings of sound (marked by triangles) and image (marked by circles). (a) VAR after the training. (b) VAR after the fine-tuning, (c) VAR++ after the training. (d) VAR++ after the fine-tuning.
4.4 EVALUATION OF THE RL POLICY
Evaluation metries. We evaluate the model with two metries: (1) success rate (SR) and (2) the number of labels used for training (LU). We define SR as the percentage of successful test episodes. We test the learned policy for 50 episodes for each intent. For the iTHOR environment, an agent succeeds if it fulfills the command. For the TurtleBot and Kuka environments, a successful episode happens when the agent stays close to the target mentioned in the command for a certain time period. We compare the label usage of a model because command following robot deployed in the real world should require as few annotations as possible from non-experts for fine-tuning.
Baselines and ablations. We compare the RL performance of our method against the following baselines and ablation models. The first baseline, denoted as "E2E," is a representative end-to-end deep RL policy for command following robots Chang et al. (020)) E2E uses hand-tuned task- specific reward functions and requires ground-truth class labels for image and sound classification.
The second bascline, denoted "VAR." trains an RL agent based on the output of the VAR Chang ctt al. (202T). VAR utilizes triplet loss for the training and fine-tuning. Both our method and VAR Eq 5 for the downstream RL tasks. We mark model with "Centered (C)" "Not centered (NC)"to "to indicate if the image and sound embeddings of the empty intent are set to the center of the hypersphere in the joint space. The original VAR method does not centralize the empty intent.
The third baseline, denoted as "ASR+NLU+RL (ANR)," is a common modular pipeline. We first an off-the-shelff automatic speech recognition (ASR) named Mozilla DeepSpeech (aanchh al. (2014) to transcribe the speech to text. We then train a lcarning-basee standing (NLU) module to handle the noisy output from the ASR. For example, "Play the music" is sometimes transcribed as "by the music." Finally, a vision-based RL agent operates with the predicted intent from the NLU. Note that unlike this baseline, our method does not rely on any tran- scriptions or expertise to be fine-tuned This baseline does not work with non-speech datasets such as NSynth. See Appendix D for more facts.
natural language under-
7

---

Under review as a conference paper at ICLR 2023
Definition of labels. In this paper, labels include all forms of annota- tion and measurement that are used to train a model. For example, one- hot labels for image and sound clas- sification and the distance measure- ment between the robot and the goal are both labels. One risual-audio pair (I, y) for training or (I, for fine- tuning used in VAR++ requires 1 la- bel to indicate or the same intent. A visual-audio triplet used in VAR, (I, ".,, requires 2 labels to in- dicate the positive and the negative. Every E2E training step requires 3 labels, including the target object state checking (e.g. check the light is switched on), distance measuring to calculate the extrinsic reward, and one-hot label for auxiliary losses.
Table 2: Test success rate results in Kuka and TurtleBot en- vironments with different sounds.
SR
Env
Dataset
ANR
E2E
VAR
Ours(C)
Wordset
85.5
95.5
97.0
99.0
Kuka
NSynth
92.5
98.0
98.0
Mix
94.0
95.5
97.0
Wordset
82.0
92.0
94.0
95.0
TurtleBot
NSynth
95.0
96.0
97.5
Mix
87.0
91.0
92.5
Control policies with unheard sounds. In this experiment, we test the performance of different models with sound com- mands never heard by the agent during training (e.g. new speakers). All the models were trained with the same num- ber of RL steps and sufficient labels. For iTHOR environment, we trained the agents for 9 million (M) RL steps and tested them within the seen floor plans (Floor Plan 201 220). For Turtlebot and Kuka environments, the total RL steps is 3M. No
Table 3:
Train label-usage and test success rate results in iTHOR 201- 220 with FSC dataset.
Models
LU (x10%) 4
SR
ANR
27.00
66.0
E2E
27.06
68.0
fine-tuning performed yet.
VAR(NC)
9.12
65.6
VAR(C)
9.12
69.0
Table 2nd and Table show that the application of our method is not limited to a specific robot, robotic task, or types of sound signal. In all environments, compared to the baselines, our method achieves the highest SR. In iTTOR environment, our method achieves the highest SR and the lowest LU. Although limit was imposed LU in this experiment, ASR+NLU+RL and E2E require much more labels during the training than VAR and our method. The results also suggest that the intrinsic rewards produced by the representations sufficient for the RL training, since VAR and our method both demonstrate satisfying performance without receiving any extrinsic rewards.
Ours(NC)
9.06
65.8
Ours(C)
9.06
72.4
From Table 2 and 3 the SR for ASR+NLU+RL baseline is lower than most of the other methods. The main reason is that the system suffers from intermediate and cascading errors among different modules, which coincides with the findings in Chang et al. (2021); Tada et al. (020). The last four rows of Table B indicate the improvement by centralizing the empty intent for both VAR and our method. This result justifies the necessity of the binary classification loss in Eq. 2 See Appendix F for examples of task execution of the agent and Appendix for time efficiency measurements.
Fine-tuning in novel iTHOR floor plans. This experiment aims to show the potential of each method to be im- proved in a new domain. We consider the scenario where trained house- hold robot is purchased to serve in a new room with unfamiliar set of fur- niture and arrangement, Each method given the same number of new labels, and a data efficient method should achieve the highest suc- cess rate. We first test the performance of trained models with unheard sound commands in 5 unseen iTHOR floor plans without any fine-tuning. This process uses new labels. The first three columns of Table show the necessity of fine-tuning: the performance of all methods drops due to the con- sequence of domain shift, which is a common problem for learning systems Tobii al.) (2017). We then use 2400 new labels for each unseen floor plan to fine-tune each method for that floor plan. For our method, each intent has 400 new labels on average because there are intents for our iTHOR environment. We followed Sec. 3.3 to fine-tune the VAR and VAR++ and used Eq. 4 to
Table 4: Average success rates over unseen iTHOR Floor
Plan 226 - 230 after fine-tuning with additional label-usage.
LU
2400
Models
ANR E2E VAR Ours
ANR E2E VAR Ours
Avg.1
18.8
18.4 19.6
20.8 24.0 23.6 69.2 86.0
8

---

Under review as a conference paper at ICLR 2023
self self-improve RL policies without current sounds. For E2E, collect one-hot labels and use simu- lator queries during the tine-tuning. The fine-tuning is terminated after it reaches the label limit. See Appendix E.4 for comparison of task execution before and after the fine-tuning
From Table we find that the ANR and end-to-end method can only be improved by 5.2% using 2400 labels, suggesting the inefficiency of fine-tuning E2E after deployment. The label quotas depleted rapidly due to the inefficient use of labels for policy network fine-tuning, which leads to less RL experience. VAR and our method improve itself by 49.6% and 65.2% respectively, using the same amount of labels after IM of self-supervised RL training steps.
The richer RL experience was due to the higher data efficiency of our method because the labels were used to update VAR++, and there was no label consumption during the self- supervised RL exploration. Com- pared to VAR, our method achieves better performance because VAR++ does not need negative pairs for fine- tuning. This property allows the VAR++ to achieve almost the same RL performance half as many labels, since since/Chang et al. (202I) reports that the SR for VAR with 5000 new labels is 84.7%.
Table 5: Model performance and the number of visual-audio pairs collected for the fine-tuning
Number of Pairs
Floor Plan
RL steps
0
600
1200
2400
OM
26.8
0.08M
41.6
70.8
72.0
84.0
226
0.4M
69.2
87.6
92.0
92.0
IM
77.6
88.4
93.2
97.2
OM
34.4
VAR using only
0.08M
51.2
54.0
54.0
64.0
229
0.4M
57.6
70.4
77.6
95.2
IM
62.0
71.2
93.2
95.6
We further show the relation between the number of RL steps and the number of newly collected pairs in two randomly selected unseen iTHOR Floor Plans (226 and 229). In Table we see that our method is still effective when the number of new pairs and the self-supervised RL steps are much fewer than 2400 and 1M- even when no new pairs are collected. More visual-audin pairs and more RL steps allow the agent to improve faster and reach higher success rates.
Fine-tuning in new Kuka environment. This experiment shows that our method can handle dy- namics gaps and adapt to unseen objects. We first train the agent the original Kuka environment with four identical blocks. test time, we change the link mass, the joint friction, and parameters of the robot's 's PID controller. In addition, as shown Fig. 7 in Appendix
C.2 we also replace three of
the blocks to capsule, a teddy bear, and rabbit. Without fine-tuning, our method achieves 69.5% SR. This result suggests that the VAR++ successfully encodes the most essential spatial information and can generalize to unseen objects with different shapes. We then fine-tune the agent following Sec. 3.3 with 1800 visual-audio pairs and 0.5M RL steps. The final SR raises to 96.5%, which demonstrates the adaptability of our method to novel objects and changes in dynamics.
FUTURE WORK AND DISCUSSION
In conclusion, propose novel visual-audio representation named VAR++ for command fol- lowing robots based on the recent advancement in (self-)supervised contrastive learning. VAR++ requires much fewer labels from non-experts during fine-tuning but produces higher-quality rewards for downstream RL agents. Our results suggest that visual-language association and skill devel- opment are highly correlated and thus need be designed together. Furthermore, we are the first to demonstrate that (self-) ysperrispee contrastive loss has the potential in enhancing human-robot interaction (HRT) experiences. Such a natural human-robot interaction can promote human percep- tion and adoption of robotic systems and marks one step towards practical social robot applications. However, our work encompasses the following limitations, which opens up directions for future work. (1) Empty intents may result in sparse intrinsic reward function, which poses challenges in solving long horizon tasks. To solve this, our reward function can be combined with other intrinsic rewards Burda et al. 2019). (2) We only apply our method to vision- based command following robots in this paper. It is a promising direction to extend the method to other modalities and provide reward function for other goal-based multi-mon modal robot tasks.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Iige Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz. Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with robot hand, arXiv preprint arXiv:1910.07113, 2019.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions real environments. In IEEE Computer Society Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 3674-3683, 2018.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations (ICLR), 2019.
Peixin Chang, Shuijing Liu, Haonan Chen, and Katherine
Robot sound interpre- tation: Combining sight and sound in learning-based control. In IEEE/RSI International Confer- ence on Intelligent Robots and Systems (IROS), pp- 5580-5587. IERE, 2020.
Peixin Chang, Shuijing Liu, and Katherine Driggs-Campbell, Learning visual-audio representations for voice-controlled robots. arXiv preprint arXiv:2109.02823, 2021.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Ra- jagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Conference on Artificial Intelligence (AAAI), pp. 2819-2826, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning (ICML). pp. 1597-1607. PMLR, 2020.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybb httpbuulet.orrg. 2016-2019.
Steven Davis and Paul Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. JEEE transactions on acoustics, speech, and signal processing, 28(4))357-366, 1980.
Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, and Deepak Pathak. Auto-tuned sim-to- real transfer. In IEEE International Conference on Robotics and Automation (ICRA), pp. 1290- 1296, 2021.
Jesse Engel, Cinjon Resnick, Adam Roberts. Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. Neural audio synthesis of musical notes with wavenet autoencoders. In International Conference on Machine Learning (ICML), pp. 1068-1077, 2017.
Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. In Robotics: Science and Systems, 2019.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint 4r1.v:1412.5567. 2014.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded lan- guage learning in simulated 3d world. arXiv preprint arXiv: 2017.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data- efficient robotic grasping via to-anoncal adaptation networks. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12627-12637, 2019.
Eric Jang, Coline Devin, Vincent Vanhoucke, and Sergey Levine. Grasp2vec: Learning object representations from self-supervised grasping. In Conference on Robot Learning (CoRI.), 2018.
10

---

Under review as a conference paper at ICLR 2023
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18611-18673, 2020.
Minjeong Kim, Gyuwan Kim, Sang- Woo Lee, and Jung- Woo Ha. St-bert: Cross-modal language model pre-training for end-to-end spoken language understanding. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 7478-7482. IEEE, 2021.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting sulf--see visual repre- sentation learning. In Conference on Computer Vision and Pattern Recognition (CVPR), pp- 1920-1929, 2019.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Sascha Lange, Martin Riedmiller, and Ame Voigtländer. Autonomous reinforcement learning raw visual input data in a real world application. In The international joint conference on neural networks (IJCNN), pp. 1-8. IEEE, 2012.
Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. Speech model pre-training for end-to-end spoken language understanding. In Annual Conference of the International Speech Communication Association (INTERSPEECI), 2019.
Aly Magassouba, Komei Sugiura, Anh Trinh Quoc, and Hisashi Kawai. Understanding natural language instructions for fetching daily objects using gan-based multimodal target-source classi- fication. IEEE Robotics and Automation Letters, 4(4):3884-3811, 2019.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Sys- tems (NeurIPS), volume 31, 2018.
Thao Nguyen, Nakul Gopalan, Roma Patel, Matt Corsaro, Ellie Pavlick, and Stefanie Tellex. Robot Object Retrieval with Contextual Natural Language Queries. In Proceedings of Robotics: Science and Systems, 2020.
Rohan Paul, Jacob Arkin, Derya Aksaray, Nicholas Roy, and Thomas M Howard. Efficient ground- ing of abstract spatial concepts for natural language interaction with robot platforms. The Inter- national Journal of Robotics Research, 37(10): 2018
J. Salamon, C. Jacoby, and J. P. Bello. A dataset and taxonomy for urban sound research. In International Conference on Multimedia (ACM-MM), pp. 1041-1044, Orlando, FL, USA, Nov. 2014.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Dmitriy Serdyuk, Yongqiang Wang, Christian Fuegen, Anuj Kumar, Baiyang Liu, and Yoshua Ben- gio. Towards end-to-end spoken language understanding. In International Conference on Acous- tics, Speech, and Signal Processing (ICASSP), pp. 5754-5758. IEEE, 2018.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. contrasttt networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation (ICRA), pp. 1134-1141. IEEE, 2018.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks, In IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition (CVPR), June 2020.
Laura Smith, J. Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In IEEE International Conference Robotics and Automation (ICRA), pp. 1-7, 2022.
11

---

Under review as a conference paper at ICLR 2023
Francesca Stramandinoli, Vadim Tikhanoff, Ugo Pattacini, and Francesco Nori. Grounding speech utterances in robotics affordances: An embodied statistical language model. In Joint IEEE Inter- national Conference Development and Learning and Epigenetic Robotics (ICDL-EpiRob), pp- 79-86, 2016.
Yuuki Tada, Yoshinobu Hagiwara, Hiroki Tanaka, and Tadahiro Taniguchi. Robust understanding of bobot-directed speech commands using sequence to sequence with noise injection. Frontiers in Robotics and AI, 6:144, 2020.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 23-30, 2017.
Andrea Vanzo, Danilo Croce, Emanuele Bastianelli, Roberto Basili, and Daniele Nardi. Robust
spoken language understanding for house service robots. Polibits, (54):11-16, 2016.
Yufei Wang, Gautham Narayan Narasimhan, Xingyu Lin, Brian Okorn, and David Held. Roll:
Visual self-supervised reinforcement learning with object reasoning. In Conference on Robot Learning (CoRL), 2020.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03220 2018.
Haonan Yu, Haichao Zhang, and Wei Xu. Interactive grounded language acquisition and general- ization in 2d world. In International Conference Learning Representations (ICLR), 2018.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision (ECCV), pp. 649-666. Springer, 2016.
12

---

Under review as a conference paper at ICLR 2023
A ALGORITHM FOR FINE-TUNING AN AGENT
This section shows the detailed algorithm for fine-tuning the VAR++ and an RL. agent.
Algorithm Fine-tumin Fine-tuning
Inputs: A trained VAR++ V, and a trained policy me
2: Collect small set of visual-audio pairs D = {I, S,)
3: for sampled minibatch {[1, S:) from 2> do
Fine-tune VAR++
4: Calculate empty intent label e, by checking if S, 0 mm
5: Calculate image and sound embeddings: h z h x t V(I,S;)
6:
Calculate Lssc by Eq. 0
Calculate loss by Lrinetuee = ai Cssc + a2 Ex= CBCE(b' (hz), eạ) + LBCE(b* es)
7:
8:
Update to minimize Lrinetune
9:
for = 0. 1, -.. do
> Self-supervised RI. fine-tuning
Sample sound command S, from D goal for & = 0, "....' do
11:
Receive RGB image I, and robot state M, Calculate image and sound embeddings: v/, vg + V(I, by Eq.
13:
14:
Calculate reward r2 v! v
15:
S, then
16:
Calculate embeddings: vē V(S,)
17:
= Ti + V Store (rr, I, M,, v v in a memory buffer RL
18:
Update Te with data from DRL using PPO
20:
Clear DRL
21:
return V, m
B SOUND DATA
Table 6: Sound signals used in the experiments.
Dataset
Sound
Examples
activate light deactivate light activate music deactivate music bring shoes
"Tur the lights." "Lamp on" "Switch off the lamp," "Lights off"
FSC
"Put on the music." "Play"
"Pause music," "Stop" "Get my shoes," "Bring shoes"
"0," "1." "2," "3" names of 4 objects
"zero," " "one," "two," "three" "house" "tree," "bird," "dog"
GSC
NSynth
C₁. E1.
Various instruments, tempo, and volume
US8K
bark, jackhammer
Sound recorded in the wild
13

---

Under review as a conference paper at ICLR 2023
C ROBOTIC ENVIRONMENT DESCRIPTIONS
The Turtlebot and Kuka environments are developed in PyBullet Coumans & Bai (016-2019) and mainly posed challenges in fine motor control with moderate difficulty in perception. contrast, the iTHOR environment is developed in A12-THOR Kolve et al. (2017) and is challenging in perception with discretized and simplified control.
C.I. TURTLEBOT
Four objects cube, sphere, cone, and cylinder are placed in 4m space. Each object has associated intent. The goal of the robot is to navigate to the object corresponding to the given sound command based on RGB images. The robot and the four objects are placed randomly in the arena at the beginning of an episode. The robot's ' action is the change of desired transitional velocity and the change desired orientation. The robot needs develop exploration skills discover the goal object in the shortest period.
Hmppyy
Empty
Figure 5: Visualization the TurtleBot environment with paired images and voices from the Wordsct. In this case, "zero" means cube, "one" means sphere, "two" means cone, and "threc" means cylinder. The red and green rays are just for illustration purposes. The right most figure shows the camera view.
C.2 KUKA
Four identical blocks, each associated with a sound command, are placed in line at a random location on the table. The robot needs move its gripper above the block corresponding to given command based on RGB images. The camera is placed at a fixed location on the side of the table such that it can capture the gripper and blocks from distorted perspective. The relative positions of the gripper tip and the blocks are initialized randomly at the beginning of episode. The robot needs to develop spatial reasoning skills to approach the target block using the relative positional information observed from the camera.
"Zasa"
Emplyy
Emppy
Fanpty
"Tero"
Camena
Figure 6: Visualization of the Kuka environment with paired images and vnices from the Wordset. In this case, "zero" means the lefimost block, "one" the second block from the left, and The red and green rays are just for illustration purposes. The right most figure shows camera view.
Figure 7: Visualization of the objects for the Kuka fine-tuning experiment.
C.3
ITHOR
Our iTHOR enviromment uses real full- sentence speech commands to simulate a real-world appli- cation of household robots. The environment has 30 different floor plans of living rooms, each with
14

---

Under review as a conference paper at ICLR 2023
their own set of decorations, furnitures, and arrangements. The robot is given goal tasks such as switching the floor lamp or television on or off. The robot must navigate through the environment and interact with the intended object given RGB images and noisy local discrete occupancy grid as robot states. The complexity of the environment requires the agent to associate complicated speech commands with high-fidelity visual observations, without floor plan map. The floor plans be visualized and interact with in https: /atps://aaithhxx ai2thor. allenai org/ /emoo
"Pense"
"Pul lhe musie""
"Tum on the lamp"
"Lights on"
"Lights off"
"Bring shoess
Figure 8: Visualization of the iTHOR environment with paired images and voices from the FSC.
D ABOUT ASR+NLU+RL (ANR) PIPELINE
Accuracy of intent prediction of ASR+NLU, FSC dataset: 86.0%; Wordset: 87.0%.
SR of the RL agent when given the ground-trute intent
iTHOR environment: 79.2%; Kuka environment: 98%; TurtleBot environment: 95%.
E VISUALIZATION OF TASK EXECUTION
E.I TURTLEBOT
Sound: Tinget vylinder
Targgtt:
Sound: C4, Inrget ouhe
Figure 9: Visualization of the task execution in the TurtleBot environment after training without fine-tuning. The sounds come from NSynth dataset. TurtleBot searches and approaches its target successfully in all episodes.
15

---

Under review as a conference paper at ICLR 2023
E.2 KUKA
Target second
Sounc: "Zeso" Target: first blok from left
JO0A
DALL
WARD
Sound: "One", Terget sccond bluck frem the left
Figure 10: Visualization of the task execution the Kuka environment after training without fine-tuning. The sounds come from Wordset dataset. Kuka moves its gripper to the target block successfully in all episodes.
E.3 ITHOR
Sound: "Bring shoes", 'arget: pick the pillow
Sound: "Switch lights", Target: turn floor lamp
Sound: "Play". Target: turn on the TV
Sound: "Stop the music". Target: off the TV
Figure 11: Visualization of the task execution in the iTHOR environment after training without fine-tuning. The sounds come from FSC dataset. iTHOR agent finishes household tasks successfully in all episodes.
16

---

Under review as a conference paper at ICLR 2023
E.4
ITHOR FINE-TUNING
Sound: "Stop the music", Target. off the TV
Before
After
Sound: "Fetch shoes", Target: Pick the pillow
Before
Afler
Sound: "Turm on the lamp". Target: Turn on the floor lamp
Before
Afler
Sound: "Turn off the lamp", Target: Turn off the floor lamp
Before
Alter
Figure 12: Visualization of the task execution in the iTHOR environment before and after the fine-tuning in floor plans and the sound commands given by speakers. The sounds from FSC dataset.
F TIME EFFICIENCY
In this section, we evaluate the time efficiency of all the methods. All the models are running on single Nvidia GTX 1080 Ti GPU and Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz. We report the average time in second (s) for the model to take one action iTHOR environment with the FSC dataset. The average is calculated from 12500 samples.
ANR: 0.041s
E2E: 0.018s
VAR: 0.024s
VAR++ 0.022s
17


Reviews:
[{"review_id": "sf0-zWWizS", "reviewer": "Reviewer_hTKz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an architecture for audio visual representations that can be used to embed multi-model representations useful for downstream robot learning tasks.", "summary_of_the_review": "Nice contribution to a very specific problem, which is clear and correct, but likely too incremental to meet the bar for a venue like ICLR.", "strength_and_weaknesses": "Strenghs:\n- very relevant problem space: this is an important area of research,\n- the model is well-motivated, and described in sufficient detail to be generally useful,\n- the quantitative results are compelling.\n\nWeaknesses:\n- The whole paper can be summarized as: 'we took VAR, replaced the triplet loss with a contrastive loss, and things worked better'. While a solid result to be publishing, the novelty is limited: the general idea that contrastive losses generally improve over triplets is well documented in the literature.\n- The model is well engineered and reasonable for the setting that's explored, but the paper doesn't provide any unique, novel insight beyond 'here is an architecture that works well for this problem'. This means there is little one will learn from reading this paper beyond this exact setting.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TyEyhEx3Oc", "reviewer": "Reviewer_Fs4r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents VAR++, that is representation that generates intrinsic rewards by associating images and sound commands. This model is used to perform command-following robot tasks. The method builds on top of VAR (Visual-audio representation), is evaluated in 3 different tasks in simulation, and is compared against other 2 baselines in addition to the basic version.", "summary_of_the_review": "The proposed method is interesting and the paper well written. The novelty and contribution of the paper are incremental. The experimental evaluation does not include enough details to be reproducible, and the quantitative evaluation is missing some results (eg relative to one of the baselines considered) and some discussion about the limitations of the proposed method.", "strength_and_weaknesses": "The paper is well structured and written in a clear way. The problem of interest is introduced and motivated in a comprehensive way, mentioning relevant literature in the field. The \"list of contributions\" reads more as the list of steps describing the approach and its advantages, rather than highlighting the novelties of this particular piece of work. The methodology is presented in a clear way and the diagrams are clarifying and well presented as well. The experimental setup is also described fairly well, however there is very little visualisation (only Fig. 4) of the simulated environment and setup. There is also no visualisation of the data and of the results obtained. The tables do report quantitative numbers, however they seem sometimes incomplete: for example why is ASR+NLU+RL not included in Table 2 and 4? Also, what is the impact of the transcription step (which is not part of the ASR+NLU+RL per se) on the performance obtained from applying ASR+NLU+RL? What are the results if you used (correct) text directly?\nThe limitations of the proposed work are not discussed. Can you describe in which cases the proposed method would fail or perform just as good as other methods? Is there an evaluation on time efficiency? Is the proposed method also more efficient with respect to other alternatives?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F867pAJLxFN", "reviewer": "Reviewer_dfAL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduced a method to learn a visual-audio conditioned control policy. The core idea is to learn a joint visual-audio representation and derive a self-supervised reward for training task policies. To acquire a high-quality visual-audio representation, they improved the existing VAR algorithm by adopting a batch-based supervised constrastive loss to improve the data efficiency. The policy reward is then defined as the difference between the embedding of the current image and the embedding of a sound signal provided at the beginning to indicate the task goal. The method was applied to two control tasks and one navigation-related task.", "summary_of_the_review": "I think the paper is in general interesting and can be a good work for the venue, though I do have a couple concerns as listed in the weaknesses above. \n\nIn addition I also have some clarification questions for the work: \n1) For E2E was it trained with 1M RL steps as well? Or it was terminated after doing a pass through the 2400 new labels?\n2) For the intrinsic reward, was eq 4 or eq 5 used in the final system? \n3) For the training process, how were the sound labels as well as intentions generated?", "strength_and_weaknesses": "Strengths:\n- Improvement over the existing VAR representation, which can be potentially helpful beyond the presented setting.\n- Self supervised RL training method appears solid and effective.\n- Enable robots to follow audio command is a useful and essential skill.\n\nWeaknesses:\n- Fine-tuning doesn’t seem very efficient in general: 2400 labels and 1M self-supervised RL training for fine-tuning seems a lot. Does the -user need to provide 2400 labels (image/audio pairs)? Providing the label and collecting 1M samples in the real-world seems infeasible.\n- Not all images have intentions in real environments, which would lead to sparse reward problems and make the method potentially difficult to handle long-range tasks.\n- No examples of handling dynamics gap were presented even though it is claimed to handle that. For robotic control tasks they are in general simple: e.g. the manipulator only needs to reach certain location without interactions like grasping. It's not clear if the method can effectively adapt to like manipulation of unseen objects.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject