A Data-Efficient Visual-Audio Representation with Intuitive Fine-tuning for Voice-Controlled Robots
Authors: Peixin Chang
Shuijing Liu
Tianchen Ji
Neeloy Chakraborty
Kaiwen Hong
Katherine Rose Driggs-Campbell
Venue: CoRL 2023 Poster (2023)
Keywords: Command Following
Multimodal Representation
Reinforcement Learning
Human-in-the-Loop

Abstract: A command-following robot that serves people in everyday life must continually improve itself in deployment domains with minimal help from its end users, instead of engineers. Previous methods are either difficult to continuously improve after the deployment or require a large number of new labels during fine-tuning. Motivated by (self-)supervised contrastive learning, we propose a novel representation that generates an intrinsic reward function for command-following robot tasks by associating images with sound commands.
After the robot is deployed in a new domain, the representation can be updated intuitively and data-efficiently by non-experts without any hand-crafted reward functions. We demonstrate our approach on various sound types and robotic tasks, including navigation and manipulation with raw sensor inputs. In simulated and real-world experiments, we show that our system can continually self-improve in previously unseen scenarios given fewer new labeled data, while still achieving better performance over previous methods.

A Data-Efficient Visual-Audio Representation with Intuitive Fine-tuning for Voice-Controlled Robots
Peixin Chang, Shuijing Liu, Tianchen Ji, Neeloy Chakraborty, Kaiwen Hong, Katherine Driggs-Campbell University of Illinois at trhaan-CChampaign (pchang17, sliu105, tj12, neeloyc2, kaiwen2, krdc ) @illinois.edu
Abstract: A command hoolwwndfollowing robot that serves people in everyday life must continually improve itself in deployment domains with minimal help from its end users, instead of engineers. Previous methods either difficult to continuously improve after the deployment require large number of new labels during fine- tuning. Motivated by (self-) suuervised contrastive learning, we propose a novel representation that generates an intrinsic reward function for command following robot tasks by associating images with sound commands. After the robot is de- ployed in a new domain, the representation can be updated intuitively and data- efficiently by non-expep non-experts without any hand-crafted reward functions. We demon- strate our approach on various sound types and robotic tasks, including navigation and manipulation with sensor inputs. In simulated and real-world experi- ments, we show that our system can continually self-improve in previously unseen scenarios given fewer new labeled data, while still achieving better performance over previous methods.
Keywords: Command Following, Multimodal Representation, Reinforcement Learning, Human-in-the-Loove
Introduction
Audio command following robots is an important application that paves the way for non-experts intuitively communicate and collaborate with robots in their daily lives. Ideally, command- following robot should ground both speech and non-speech commands visual observations and motor skills. For example, a household robot must open the door when it hears doorbell some- one saying "open the door." The robot should also be customizable and continually improve its interpretation of language and skills from non-experts [1, 2].
The need for command-follee followid robots has spurred wealth research. Learning- based language grounding agents were proposed to perform tasks according visual observations and text/speech instructions [3, 6, 7]. However, these approaches often fail to completely solve common prob- lem in learning- based methods: performance degradation in novel target domain, such the real world [8, 9, 10]. Fine-tuning the models in the real world is often expensive due to the requirement of expertise, extra equipment and large amounts of labels, none of which can be easily provided by non-expert users in everyday environments. Without enough domain expertise or abundant labeled datu, how can we allow users to customize such robots to their domains with minimal supervision? Some prior works have attempted to reduce data usage when fine-tuning the robot in new domains. However, the efficiency of the methods usually relies task-specific assumptions [11], extra sensor instrumentation [12], and limited task variations [13, 14].
In this paper, we propose a novel framework that builds on keeff-)superviie contrastive learning to realize more effective training and more efficient fine-tuning for rewards and skills learning. As shown in Fig. 1, we first learn joint Visual and Audio Representation, which Data-efficient and can be Intuitively (Dif--AA In the second stage, we use the representation compute
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

by ongineers
Sewerakora
Improve by users
Turn on the TV
Switch the TV
Haai
Date
Dif-VAR
"Lights
"Tam the lamp?
Light
Rewards
Sell Sellssperiiedd skill Improvement
Figure 1: Our pipeline. Contrastive learning used to group images and audio commands of the same intent. The resulting Dif-VAR supports the downstream RL training by encoding the auditory and visual signals and providing reward signals and states to the agent. Users improve the Dif- VAR intuitively by providing the visual- audio pairs in their domain, and the updated Dif-VA VAR supervises the RL fine-tuning.
intrinsic reward functions learn various robot skills with reinforcement learning (RL) without any reward engineering. When the robot is deployed in new domain, the fine-tuning stage is data efficient in terms of label usage and is natural to non-experts, since users only need to provide relatively small number of images and their corresponding sounds. For example, a user can teach Dif- VAR by saying that "this an apple" when the robot sees an apple. Then, RL policies are self-improved with the updated Dif- VAR.
We apply this learning approach to diverse navigation and manipulation environments. Given sound command, the robot must identify the commander's goal (intent), draw the correspondence between the raw visual and audio inputs, and develop a policy to finish the task. To develop gener- ally applicable pipeline, we make minimal assumptions. We do not assume the availability of human demonstrations, prior knowledge about the environment, or image and sound recognition modules that have perfect accuracy after the deployment. The robot equipped with only monocular uncal- ibrated RGB camera and microphone. Our method works with various sound signals (e.g. voice, environmental sound) and various types of robots (e.g. mobile robots, manipulators).
Our main contributions are: (1) We propose voice-controlled robot pipeline for everyday house- hold tasks. This framework intuitive for non-experts to continually improve and is agnostic to a wide range of voice-controlled tasks. (2) Inspired by (supe-)supervised contrastive loss, we propose novel representation of visual- observations named Dif- VAR, which generates intrinsic RL rewards for robot skill learning. Only image-audio pairs are required to fine-tune Dif- Dif-VAR and the RL policy. No reward engineering, state estimation, other supervision is needed. (3) We pro- pose a variety of voice-controlled navigation and manipulation benchmarks. In new simulated and real-world domains, the adaptation of our method outperforms state-of-the-arr basclines in terms of performance and data efficiency, while requiring less expertise and environmental instrumentation.
2 Related Works
End-to-end language understanding. End-to-end spoken language understanding (SLU) sys- tems extract the speaker's intent directly from raw speech signals without translating the speech to text [15, 16, 17]. Such an end-to-end system is able to fully exploit subtle information, such as speaker emotions that are lost during speech-to-text transcription, and outperform pipelines that preprocess the speech into text [16, 17). However, they have not been widely applied in robotics.
Command following agents. Conventional command following agents consist of independent modules for language understanding, language grounding, and planning [18, 19, 20]. But these modular pipelines suffer from intermediate errors and do not generalize beyond their programmed domains [21, 22, 23]. To address these problems, end-to-end command following agents are used to perform tasks according to text-based natural language instructions and visual observa- tions [3, 4, 23, 24, 25]. In addition, large language models are applied to program the robots given the language prompts [26, 27J. However, these methods neglect the non-speech commands and abstract away the practical challenges of auditory signal grounding. To make full use of audio com- mands, Chang et al. [7] introduces an RL framework for skill learning directly from raw sounds.

---

However, all these works overlook the continual fine-tuning and customization by users, an essen- tial step to ensure performance after the deployment. Fine-tuning such models is computationally challenging and requires hand-tuned reward functions and prohibitive labeling efforts. Chang et al. [1] partially addresses the problem by learning a visual-audio representation (VAR) with triplet loss to generate an intrinsic reward function for RL. However, this method requires negative pairs in the triplet loss, which is not intuitive to non-experts and inefficient to deploy in new target domains.
Visual and language representation for robotics. Representation learning has shown great poten- tial in learning useful embeddings for downstream robotic tasks [28, 29). Deep autoencoders have been used to learn latent space which generates states or rewards for RL [30, 31, 32]. Contrastive learning has also been used to learn representations for downstream skill learning [33, 34). However, in task execution, goal image has to be provided, which is less natural in terms of human-robot communication compared to language.
To this end, visual-language representations have been widely used to associate human instructions with visual observations in navigation [27, 35] and manipulation [12, 36, 37J. However, similar to text-based command following agents, these methods also lose information when the input is sound instead of text. Although audio-visual representations such as AudioCLIP have been developed [38], how to apply them in robot learning remains an open challenge. Our work and [1] address this chal- lenge by proposing visual-audio representation that generates RL rewards for robot skill learning, while our method achieves better data efficiency and can be more easily fine-tuned than [1].
3 Methodology
In this section, we describe the two-stage training pipeline and fine-tuning procedure. In training, we assume the availability sufficiently large labeled datasets, simulators, and labels. However, in fine-tuning, speech transcriptions, one-hot labels, and reward functions, are not available.
3.1 Visual-audio representation learning
In the first stage, we collect visual-audio pairs from the environment. Then, we learn a joint repre- sentation of images and audios that associates an image with its corresponding sound command.
Data collection. Suppose there are MM possible intents or tasks within an environment. We collect visual-audio pairs defined as (I, S, y) from the environment, where I E R=** is an RGB image from the robot's camera, S E IR xxm is the Mel Frequency Cepstral Coefficients (MFCC) [39] of the sound command, and y E {0, 1.....,.I } is the intent ID. We call I and S two views of an intent y. visual-audio pair contains a goal image and a sound command of the same intent. For example, when an iTHOR agent sees lit lamp, the agent hears the sound "Switch on the lamp" from the environment. In contrast, when the agent does not any object in interest or is far away from all objects so that it sees multiple objects at once, it receives only image and hears sound. The image is paired with $ = 0/xm and y = M. We define this situation as empty intent.
Training Dif-VAR. Our goal is to encode both visual and auditory signals into a joint latent space, where the embeddings from the same intents are pulled closer together than embeddings from dif- ferent intents. For example, the embedding of an image with TV turned on needs to be close to the embedding of sound command "Tum the TV" but far away from other irrelevant com- mands such as "TTrn off the light." We adopt the idea from (self-)supervised contrastive learning for visual representations and formulate the problem as metric learning. As shown in Fig. 2a, the VAR is a branch network with two main components. The first component contains the encoders fr R=xn + d and fs R'xm - Is which map an input image I and a sound signal S to representation vectors hn and hS respectively. In practice, any deep models for image and sound processing can be used for fl and fs. The second component is the projection heads g¹ Rdd + Rdd g. ds Rdd and b' Bd R that map the representations h' and h to the space where losses are applied. We denote the vector embeddings g' (h') and وو (hh) as z and z
3

---

Actinn
A
2968
SkI13
IN
4/
DIY-AR
Enooder
Projection Heads
Figure Purposed framework. (a) The DiR-VAR is double-branch network optimized with (self-) (spp-)sseeeeeeed contrastive loss. (b) The latent space of the DiR-VAR is unit hypersphere such that the images and audios of the same intent are closer than those of different intent in the space. (c) The Dif- VAR decides which skill to activate according to Sg.
respectively. We enforce the norm of z' and z to be by applying an L2-normalization, such that the embeddings live on a unit hypersphere as shown Fig. 2b.
We supervised contrastive (SupCon) loss as the objective, which encourages the distance be- tween z' and 2 of the same intent to be closer than those of different intent [40]. Suppose there are N visual-audio pairs in a batch. Let k K == {1, 2N) be the index of an image sound signal within that batch and P(k) = {p E K {k} Up = y be the set of indices of all images and sounds of the same intent except for index k. Then, the SupCon loss is
exp (Zk- Zp/T)
SupConn
Σ REK
log Σ [jE((AA exp (Zk: 2,/T)*
P(k)
PEP(K)
where the cardinality, z" can be either z' or z², and E R a scalar temperature parameter. The use of SupCon loss allows attraction and repulsion among all images and sound within a batch, which improves the training efficiency of the representation. We introduce binary classification loss for the image to distinguish between empty and non-empty intent. Let LICF denote the binary cross entropy loss and denote the label of intent, which is for empty intent and 1 for non-empty intent. The batch loss for training the Dif-VAR is:
N
1
LDRAVAR = (r) CSupCon + 02
Σ 1=1
Lnc ez)
(2)
N
where 0rj and 02 are the weights of losses. Depending on if the intent is predicted empty not, the output v and S of DiR-VAR can be determined for image and sound by:
v (aa 205) z', b'=b'(h')
= 11 (s.
where 1 is an indicator function. The purpose of the binary classification to set the image and sound embeddings of the empty intent to the center the joint latent space. This centralization removes the biases caused by the location of the empty intent in the joint latent space, leading to better intrinsic reward then previous methods.
While SupCon loss and other self-supervised visual representation learning frameworks are origi- nally only applied to image modality [40, 41], we extend the framework to a multi-modality setting and create a new representation for command following robots.
3.2 RL with visual-audio representation
The second stage of our pipeline is to train an RL agent using an intrinsic reward function generated by trained VAR. We model robot command following task as Markov Decision Process (MDP), defined by the tuple (X, A, P. r). At each time step t, the agent receives image

---

I from its RGB camera, and robot states M, such as end-effector location or previous action.
At = 0, an additional one-time sound command S containing an intent is given to the robot. We freeze the trainable weights of Dif-VAR in this stage and define the MDP state 22 E X as Ft = [r, v v.S. gg" MrJ, where v and v are the output of the Dif- VAR for I, and Sy, respectively. Then, based on its policy T()r(2)), the agent takes an action at E A. In return, the agent receives unknown state transition
reward rt R and transitions to the next state 2141 according P(- 4z). The process continues until & exceeds the maximum episode length T, and the next episode starts.
Intrinsic rewards. Since v' and v of the same intent are pulled together within the DiR-VAR by the contrastive loss, intrinsic rewards can be derived as the similarity between v' and v Eq. 4 and 5 present two possible task-agnostic and robot-agnostic reward functions:
r
v-=v/-ve + (5)
where vi is the embedding of the current sound signal S,, which can be triggered in the same way as S as described in Section 3.1. Intuitively, the agent using ri receives high reward when the scene sees matches the command it hears. The agent trained using the reward rf additionally needs to match the current sound it hears with the sound command to receive high rewards, Compared to ry the reward function ri does not depend on any real-time supervision signal such as current sound v from the environment, allowing the agent to perform self-supervised RL training with Dif- VAR. Although RL agents trained with Eq. 4 can already achieve decent performance, providing the current sound S, can further improve the performance [1]. Since S, can be difficult to obtain especially in real environments, S; is not part of the state 2, and thus the robot policy does not require S, at test time.
Policy network. We purpose two architectures for RL policies. The first architecture is flat and uses single policy network fulfill all the intents in environment, which is suitable when the skills to finish tasks are similar. The second architecture is hierarchical and contains multiple policy networks individually designed to fulfill a subset of intents in an environment, as shown in Fig. 2c. Given an Sy and a set of policies IT = {mi, -2, ....... the DiR selects a policy πī by
V. C;
L()..
arg max
(6)
[Cal12
where C, is the centroid of an intent in the joint latent space calculated from the training data and a lookup table that maps an intent ID to a policy. For benchmarking purposes, we use Proximal Policy Optimization (PPO) for policy and value function learning [42].
3.3 Intuitive and data-cfficient fine-tuning
After the robot is deployed in new domain such as the real world, its performance often degrades due to domain shift from both perception and dynamics [43]. Our fine-tuning procedure allows non-experts to continually improve the DiR-VAR to reduce perception gaps and improve robot skills to reduce dynamics gaps. We only need to collect visual-audio pairs of the form SI from non- experts. Since we no longer have the underlying labels y for images and sounds, we replace the SupCon loss in Eq. (2) with the following self-supervised contrastive loss (SSC) [41]:
exp (Zy ZT(K:/T)
Lssc = =- -' log ΣtEK(()) exp (zy Zz/T)²
KEK
where p(k) is the index of the data paired with the data of index k with the same intent. We mix the new data from the non-experts with a subset of the original training data to update the Dif- VAR, producing a more accurate reward function by Eq. 4. The robot can then self-improve its policy network with the reward function by randomly sampling a sound command as the goal. The collection of the visual-audio pairs does not require special equipment other than an RGB camera and microphone. The users provide images and sound based on their common knowledge using their own voices. The users do not need to type in speech transcriptions, draw bounding boxes and
5

---

masks, modify the network architectures, design reward function. To fine-tune VAR in [1], non-experts have to provide a sound command with different intent S for each image to use triplet loss. In contrast, MAR eliminates this requirement by utilizing the SSC, leading to a more intuitive data collection experience for non-experts and better performance with fewer labeled data. See Appendix for the fine-tuning algorithm.
Experiments
In this section, we first describe the various environments and sound datasets. Then, compare the performance and data efficiency of our pipeline with several baselines and ablation models.
4.1 Environments and sound dataset
Robotic environments: We evaluate the perfor- mance of all the methods on three different robotic environments: iTHOR, Desk, and Row. In all envi- ronments, the perception of the robot comes from a monocular uncalibrated RGB camera, and the robot must fulfill the sound command. See Appendix for details.
ITHIOR
Desk
Row
Figure 3; Simulation environments.
Sound data: We use several types of sounds from state-of-the-art datasets in training and testing. Specifically, we use speech signals from Fluent Speech Commands (FSC) [16] and short speech commands from Google Speech Commands (GSC) [44]. We also collect a synthetic speech dataset using Google Text-to-Speech. We use single-tone signals from NSynth [45] and environmental sounds from UrbanSound8K (US8K) [46] and ESC-50 [47]. The Wordset dataset was created from the "0,"" "2," "3" in GSC. We also used Mix datasel to show that the Dif-VAR can map multiple types of sounds to a single object or idea, by mixing speech data with environmental sound. See Appendix C for more sound examples and intent we choose for the environment.
4.2 Evaluation of the RL policy
Evaluation metrics. We evaluate the model with two metries: (1) success rate (SR) and (2) the number of labels used for training (LU). We define SR the percentage of successful test episodes. We test the learned policy for 50 episodes for each intent across multiple random seeds, and an agent succeeds if it fulfills the command. We compare the label usage of the models because command following robot deployed in the real world should require as few annotations as possible from non-experts for fine-tuning.
Baselines and ablations. We compare the RL performance of our method against the following baselines and ablations. (1) "E2E" is representative end-to-end deep RL policy for voice-controlled robots [7]. E2E uses hand-tuned task-specific reward functions and requires ground-truth class la- (2) "VAR" trains an RL agent based on the output of the VAR [1]. VAR utilizes triplet loss for training and fine-tuning. Both our method and VAR use Eq for the downstream RL tasks. (3) We compare the performance between flat (F) and hierarchical (H) architectures that have the same total trainable parameters. (4) "ASR+NLU+RL (ANR)" is common modular pipeline. Note that, unlike this baseline, our method does not rely on any tran- scriptions expertise to be fine-tuned. This baseline does not work with non-speech datasets such as NSynth, which will be indicated by "_" (5) "CLIP" uses ASR for speech recognition and uses the dot product of the embeddings from the CLIP model as the reward [48]. We use "CLIP" as rep- resentative of pre-trained visual-language models that claim zero-shot transferability to downstream tasks [48]. (6) "Oracle" is an RI. agent which assumes perfect ASR and NLU modules and is trained with hand-tuned reward functions and ground-truth class labels. See Appendix for more details.
bels for image and sound classification.
Definition of labels. In this paper, labels include all forms of annotation and measurement that used to train model. For example, one-hot labels for image and sound classification and the

---

Table Test success rate with various types of sound commands in the original visual domain.
SR
Env
Steps
Dataset
(x109)
CLIP
ANR
E2E
VAR
Ours(F)
Ours(H)
Oracle
3.0
Wordset
1.5
85.5
95.5
97.0
98.0
96.0
98.0
Row
3.0
NSynth
92.5
98.0
98.0
97.0
98.0
3.0
Mix
94.0
95.5
97.0
95.0
98.0
Desk
9.0
Mix
77.0
58.5
84.5
89.5
90.0
iTHOR
9.0
FSC
10.8
66.0
68.0
65.6
72.4
76.8
79.2
distance measurement between the robot and the goal are both labels. One visual-audio pair (I, y) for training or (I, S) for fine-tuning used in DiR-VAR requires 1 label to indicate the same intent. A visual-audio triplet used in VAR, (I, S. S,,, requires labels to indicate the positive and the negative. Every F2E training step requires about 3 labels, including the target object state checking (e.g. check if the light is switched on), distance measuring to calculate the extrinsic reward, and one-hot label for auxiliary losses.
Control policies with unheard sounds. In this experiment, we test the performance of different models with sound commands never heard by the agent during training (e.g. new speakers). All the models are trained with the same number of RL steps and sufficient labels. They are tested in the original Floor Plans 201-220 or desk. No fine-tuning is performed yet.
Table suggests that the intrinsic rewards produced by our representation adequately support the RL training across various robots, robotic tasks, and types of sound signals. Remarkably, our method demonstrates satisfactory performance from severe domain shift problem in our task. As aeneral-purpose pretrained model, CLIP is not tailored to generate RL reward or designed for any specific downstream robotic applications. ANR is limited to speech signals and has lower SR than the other methods due to intermediate error, which coincides with the findings in [1, 22j. Compared to our method, the VAR does not perform well in every environment, which suggests that the Dif-VAR produces better representation and more reliable rewards. The flat architecture outperforms the hierarchical architecture in only the Row environment, indicating that the hierarchical architecture is more suitable when the skills required to complete tasks vary. See Appendix D for examples of task execution of the agent.
without the inclusion of extrinsic rewards. CL.IP suffers
Fine-tuning in novel domains. This ex- periment aims to show the potential each method to be improved in new do- main. We consider the scenario where a trained household robot is purchased serve in new place. Each method is given the same number of new labels, and a data-efficient method should achieve the highest success rate. We first test the per- formance of trained models with unheard sound commands in unseen domains with- Ours(H) 23.2
Table Average success rates over unseen domains before and after fine-tuning.
iTHOR(sim)
Desk(sim)
Row(real)
LU
0
253
150
300
ANR
18.8
19.8
13.8
15.0
E2E
18.4
19.8
44.5
45.0
15.0
15.0
VAR
19.6
57.9
35.5
58.0
18.8
56.3
Ours(F)
20.8
85.8
69.0
84.5
18.8
78.8
88.4
70.0
90.0
16.3
75.0
any fine-tuning. For the iTHOR environment, the agent is tested in Floor Plans 226-230, which have sets of furniture and arrangements that are unfamiliar to the robot. For the Desk environment, the robot is placed in front of new desk with unseen object appearances and locations. The results are marked by "LU=0" in Table as 0 new label is required for the test. We see that the perfor- mance of all methods drops compared to the test results the original domains shown in Table 1. This phenomenon is due to the common problem domain shift faced by learning systems [8]. We then manually collect an average of 253 new labels for each unseen floor plan to fine-tune each method for that floor plan in the iTHOR environment, and 150 new labels for the unseen desk in the Desk environment. We followed Sec. 3.3 to fine-tune the VAR and Dif- VAR and used Eq. to self-improve RL policies without current sounds for 1M timesteps. For E2E, we collect one-hot
7

---

labels and use simulator queries during the fine-tuning The fine-tuning is terminated after it reaches the label limit. See Appendix 1).4 and D.5 for task execution hefore and after the fine-tuning.
From Table we find that the ANR and E2E can only be improved by less than 1.5%, suggesting the inefficiency of fine-tuning these methods after deployment. The label quotas are depleted rapidly due to the inefficient use of labels for policy network fine-tuning, which leads to less RL experience. Our methods have higher data efficiency as labels were purely used to update the VAR, and there was no label consumption during the self- supervied RL exploration. This leads to an overall richer RI experience. Compared to VAR, our method achieves better performance using the same number labels because DiR-VAR does not need negative pairs for fine-tuning. Using around 250 image-sound pairs, our method successfully improves itself fulfill 4 ~ 5 tasks in a new domain. We believe the effort manageable by non-expert. See Appendix E.6 for intermediate results versus the label usage during fine-tuning.
Fine-tuning in the real world. This experiment shows that our method is practical and helps minimize the sim2real gap. this experiment, the agent performs grasping task with noisy background using single un- calibrated RGB camera. This setting
Simulation
Bocl-woorldsctup
Camera
Figure 4: Sim2real experiment setup.
is challenging for user-involved sim2real because the performance of the models is sensitive to the inevitable inconsistency in the camera pose between simulation and the real world [1], and is un- likely that non-experts know how to calibrate a camera. To solve the problem, the DiR-VAR learns to associate valid pre-grasp poses with images and speech commands. Dif- VAR outputs high reward when the robot reaches the desired object with correct grasping pose. We first train the agents with domain randomization [8] in the simulator. Then, deploy the model to real Kinova-Gen3 arm and perform 20 tests for each intent (80 in total). See Fig. 4 and Appendix B.2 for more details. We spent an hour collecting the iisual-audio pairs and fine-tuning the policy. The last two columns of Table 2 shows that our methods minimized the domain shift with a reasonable number of newly provided pairs. Qualitative results are shown in Fig. and the supplementary video.
Sound: "Would you mind giving me the fourth object", Target: Pick up the rightmost object
After Before
Figure 5: Before and after the fine-tuning in the real world.
5 Conclusion, Limitations and Future work
In conclusion, we propose a novel visual-audio representation named Dif-VAR for command fol- lowing robots based on the recent advancement in (self- superrieee contrastive learning. Dif-VAR requires much fewer labels from non-experts during fine-tuning but produces higher-quality rewards for downstream RL agents. Our results suggest that visual-language association and skill develop- ment are highly correlated and thus need to be designed together. Furthermore, we are the first to demonstrate that ssuperiidd contrastive loss has the potential to enhanee human-robot inter- action. However, First, empty intents may result in sparse intrinsic reward functions, which pose challenges in long- horizon tasks. Our reward function can be combined with other intrinsic rewards [49]. To increase robustness, we may take 3D information and history into reward calculation. Second, user studies are needed to confirm that collecting visual-audio pairs is intuitive and convenient for end-usen indussers. Third, our method can be combined with imitation learning to further improve data efficiency.
work has the following limitations, which open up directions for future work.
8

---

Acknowledgments
This work supported by AIFARMS through the Agriculture and Food Research Initiative (AFRI) grant 2020-67021-3279//prieeer accession no.1024178 from the USDA National Institute of Food and Agriculture. We thank Yunzhu Li and Karen Livescu for insightful discussions and all reviewers for their feedback.
References
[1] Chang, S. Liu, and K. Driggs-Campbell. Learning visual-audio representations for voice- controlled robots. In IEEE International Conference Robotics and Automation (ICRA), 2023.
[2] C. Matuszek. Grounded language learning: Where robotics and nlp meet. In International Joint Conference on Artificial Intelligence (IJCAI), 2018.
[3] Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, Gould, and A. van den Hengel. Vision-and aodoo--ndlanguagg navigation: Interpreting Assually-grounde naviga- tion instructions in real environments. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3674-3683, 2018.
[4] M. Shridhar, J. Thomason, D. Gordon, Bisk, W. Han, R. Mottaghi, Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[5] V. Blukis, C. Paxton, D. Fox, A. Garg, and Y. Artzi A persistent spatial semantic representa- tion for high-level natural language instruction execution. In Conference on Robot Learning (CoRL), pages 706-717, 2022.
[6] S. Y. Min, D. S. Chaplor, P. Ravikumar, Y. Bisk, and R. Salakhutdinov. Film: Following instructions in language with modular methods. arXiv preprint arXiv:2110.07342 2021.
[7] P. Chang, S. Liu, H. Chen, and K. Driggs-Campbell Robot sound interpretation: Combining sight and sound in learning- based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5580-5587, 2020.
[8] Tobin, Fong, A. Ray, J. Schneider, W. Zaremba, and Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 23-30, 2017.
[9] James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via wwwmiiiee--o-caaoniee adaptation networks. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12627-12637. 2019.
[10] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, Tworek, Welinder, L. Weng, Q. Yuan, Zaremba, and L. Zhang. Solving rubik's 's cube with a robot hand. arXiv preprint arXiv: 1910,0710...... 2019.
[11] B. Wen, W. Lian, K. Bekris, and S. Schaal. You only demonstrate once: Category-leve ma- nipulation from single visual demonstration. In Robotics: Science and Systems (RSS), 2022.
[12] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. Conference on Robol Learning (CoRL), 2022.
[13] A. Yu and R. J. Mooney, Using both demonstrations and language instructions to efficiently learn robotic tasks. In International Conference Learning Representations (ICLR), 2022.

---

[14] M. Du, O. Y. Lee, S. Nair, and C. Finn. Play it by ear. Learning skills amidst occlusion through audio-visual imitation learning. In Robotics: Science and Systems, 2022.
[15] D. Serdyuk, Y. Wang. C. Fuegen, A. Kumar, B. Liu, and Y. Bengio. Towards end-to-end spoken language understanding. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 5754-5758, 2018.
[16] L. Lugosch, M. Ravanelli, P. Ignoto, V. Tomar, and Y. Bengio. Speech model pre-training for end-to-end spoken language understanding. In Annual Conference of the International Speech Communication Association (INTERSPEECH), 2019,
[17] M. Kim, G. Kim, S.-W. Lee, and J.-W. Ha. St-bert: Cross-modal language model pre-training for end-to-end spoken language understanding. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 7478-7482, 2021.
[18] F. Stramandinoli, V. Tikhanoff, U. Pattacini, and F. Nori. Grounding speech utterances in roboties affordances: An embodied statistical language model. In Joint IEFE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), pages 79-86, 2016.
[19] R. Paul, J. Arkin, Aksaray, N. Roy, and M. Howard. Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms. The International Journal of Robotics Research, 37(10): (000)):2199999999 2018.
[20] A. Magassouba, K. Sugiura, A. T. Quoc, and H. Kawai. Understanding natural language instructions for fetching daily objects using gan-based multimodal target-source classification. IEEE Robotics and Automation Letters, 4(4):3884-388 2019.
[21] A. Vanzo, D. Croce, E. Bastianelli, R. Basili, and D. Nardi. Robust spoken language under- standing for house service robots. Polibits, (54):11-16 2016.
[22] Y. Tada, Y. Hagiwara, H. Tanaka, and T. Taniguchi. Robust understanding of robot-directed speech commands using sequence to sequence with noise injection. Frontiers in Robotics and AI. 6:144, 2020.
[23] K. M. Hermann, F. Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. M. Czarnecki, M. Jaderberg, D. Teplyashin, et al. Grounded language learning in simulated 3d world. arXiv preprint arXiv: Skiv:1700.0655/// 2017.
[24] H. Yu, H. Zhang, and W. Xu. Interactive grounded language acquisition and generalization in a 2d world. In International Conference on Learning Representations (ICLR), 2018.
[25] D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, and R. Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Conference on Artificial Intelligence (AAAI), pages 2819 2819-2826, 2018.
[26] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho. J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, Levine, Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, Rettinghouse, D. Reyes, Sermanet, N. Sievers, C. Tan, Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng. Do as can and not say: Grounding language in robotic affordances. In arXiv preprint 12X:::224.00691, 2022.
[27] C. Huang, O. Mees, Zeng, and W. Burgard. Visual language maps for robot navigation. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
[28] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero task generalization with robotic imitation learning. In Conference on Robot Learning (CoRL), pages 991-1002, 2022.
10

---

[29] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. In Conference on Robot Learning (CoRL., 2022.
[30] A. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
[31] Y. Wang. G. N. Narasimhan, X. Lin, B. Okom, and D. Held. Roll: Visual self-supervised reinforcement learning with object reasoning. In Conference on Robot Learning (CoRL), 2020.
[32] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine. The in- gredients of real world robotic reinforcement learning. In International Conference on Learn- ing Representations (ICLR), 2020.
[33] Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, Levine, and G. Brain, Time- contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation (ICRA), pages 1 1134-1141, 2018.
[34] E. Jang. C. Devin, Vanhoucke, and Levine. Grasp2vec: Learning object representations from self-supervised grasping. In Conference on Robot Learning (CoRL), 2018
[35] D. Shah, B. Osiński, brian ichter, and S. Levine. LM-nav: Robotic navigation with large pre- trained models of language, vision, and action. In Conference on Robot Learning (CoRL), 2022.
[36] A. Yu and R. Mooney. Using both demonstrations and language instructions to efficiently learn robotic tasks. In Conference on Robot Learning (CoRL), 2023.
[37] O. Mees, Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
[38] A. Guzhov, F. Raue, J. Hees, and A. Dengel. Audioclip: Extending clip image, text and au- dio. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 976-980, 2022.
[39] S. Davis and P. Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE transactions acoustics, speech, and signal processing, 28(4):357-366, 1980.
[40] Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, Isola, A. Maschinot, C. Liu, and Kr- ishnan. Supervised contrastive learning. Advances Neural Information Processing Systems, 33:18661-18673, 2020.
[41] T. Chen, S. Korbllith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning (ICML). pages 1597-1607, 2020.
[42] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: / 1707.06347, 2017.
[43] Y. Du, O. Watkins, T. Darrell, P. Abbecl, and D. Pathak. Auto-tuned sim-to-real transfer. In IEEE International Conference on Robotics and Automation (ICRA), pages 1290-1296, 2021.
[44] P. Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv: 2018.
[45] Engel, C. Resnick, A. Roberts, S. Dieleman, D. Eck, K. Simonyan, and M. Norouzi. Neural audio synthesis of musical notes with wavenet autoencoders. In International Conference Machine Learning (ICML), pages 1068

---

[46] J. Salamon, C. Jacoby, and J. P. Bello. A dataset and taxonomy for urban sound research. In International Conference on Multimedia (ACM-MM), pages 1041-1044, 2014.
[47] K. J. Piczak. ESC: Dataset for Environmental Sound Classification. In Annual ACM Confer- ence Multimedia, pages 1015-1018. ACM Press, 2015.
[48] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, Askell, Mishkin, Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 8748-8763. PMLR, 2021.
[49] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. International Conference on Learning Representations (ICLR), 2019.
[50] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybullet org, 2016-2019.
[51] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L.. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. A12-THOR: An Interactive 3D Environment for Visual AL arXiv, 2017.
[52] O. Mees, Hermann, E. Rosete-Beas and Burgard. Calvin: benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Roboties and Automation Letters (RA-L), 7(3):7327-7334, 2022.
[53] A. Hannun, C. Case, Casper, B. Catanzaro, G. Diamos, Elsen, R. Prenger, S. Satheesh, Sengupta, A. Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv: 2014.
[54] T. Bunk, D. Varshneya, V. Vlasov, and A. Nichol. Diet: Lightweight language understanding for dialogue systems. arXiv preprint arXiv:2004.09936 2020.
12

---

A Algorithm for fine-tuning an agent
Algorithm Fine-tuning the Dif- VAR (F) and an RL agent
1: Inputs: A trained Dif ViR V, trained policy me subset of the original training data Dold
2: Collect a small set of visual-audio pairs D {(I, S.))
3: Dnew Dota UD
4: for a sampled minibatch {(I, S from Dnew do
> Fine-tune Dif- VAR
5: Calculate empty intent label 6; by checking if S; 0 m
6: Calculate image and sound embeddings: h z h x + V(I.S))
7: Calculate Lssc by Eq. 7
8:
Calculate loss by Liinetune on Lssc + 02 Ej= LaCE(Dl (h)).eg
9:
Update V minimize Ltinetune
10:
for = 0, .... do
Self-supervised RL fine-tuning
11:
Sample sound command Sy from D goal
12:
for = 0, ...., T do
13:
Receive RGB image I, and robot state M,
14:
Calculate image and sound embeddings: v v +- V(I, by Eq.
15:
Calculate reward = v! v
if S, then
17:
Calculate embeddings: vs + V(S.)
r; = Tr V S
19:
Store (rt, I, M,, v v in memory buffer Dn
20:
Update with data from D using PPO
21:
Clear RL
22:
return V, m8
B
Robotic environment descriptions
The Row and Desk environments are developed in PyBullet [50] and focus mainly on manipulation tasks. In contrast, the iTHOR environment is developed in A12-THOR [51] and is challenging in perception and designed for mobile robots.
B.1 Row
Four objects placed in a line at random location unknown to the robot on the table. A robot arm needs to move its gripper and stay above the object corresponding to given command based on RGB images. The camera is placed at a fixed location on the side of the table such that it can capture the gripper and the objects from a distorted perspective. The relative positions of the gripper tip and the objects are initialized randomly the beginning an episode. only mentions the orindal information about the target object, and the robot needs to develop spatial reasoning skills approach the target object using the relative positional information observed from the camera.
sound command
"Zoro"
Empty
Emply
Empty
"Zero"
Comem
Figure 6: Visualization of the Row environment using Kuka-iiwa robot with paired images and voices from the Wordset. In this case, "zero" means leftmost block, "one" means second block from the left, and so on. The red and green rays are just for illustration purposes. The possible locations of the blocks are limited to the green rectangle and the end- and-effector location is indicated by the vertical ray. The rightmost figure shows the camera view.
13

---

B.2 Row real
This environment modified from the original Row environment. The Four objects are mug, soup can, a pudding box, and an orange. Different objects may require distinct grasping poses. See Fig. for examples. the end of an episode, the gripper performs grasp by lowering its height from its current position, closing the fingers, and lifting the object up. For domain randomization, randomize the background, camera viewpoint, and relative offset among the objects.
"Hand over the fourth object"
"Give me the Jeftmost object"
"Bring the third object right
"Bring me the third object left"
"Give the rightmost objectt
Figure 7: Visualization of the Row environment with paired images and voices from the Synthetic dataset under domain randomization setting. The grasping poses the mug and the pudding box different. The and green rays are just for illustration purposes. The rightmost figure shows the camera view.
B.3
Desk
The Desk environment is modified from the CALVIN dataset [52]. A Franka Panda robot arm is placed in front of a desk with a sliding door and a drawer that can be opened and closed. On the desk, there is a button connected to an alarm clock, switch to control light bulb, and pill case. The tasks of the robot include turning on or off the light bulb by manipulating the switch, pressing button to mute the alarm clock and turn the LED of the clock into red, and picking up the pill case that could be on the top of the desk inside closed drawer. When the pill case is located inside closed drawer, the robot needs open the drawer before picking up the pill case. The sound commands come from FSC, ESC-50, and the Synthetic dataset.
"Lights
Empty
"Lights ofl"
"Give the
Figure 8: Visualization of the Desk environment with paired images and voices. The rightmost figure shows the camera view.
B.4 iTHOR
Our iTHOR environment uses real full-sentence speech commands to simulate real-world applica- tion household robots. The environment has 30 different floor plans living rooms, each with its own set of decorations, furniture, and arrangements. The robot given goal tasks such as switch- ing the floor lamp or television on off. The robot must navigate through the environment and interact with the intended object given RGB images and noisy local discrete occupancy grid the robot states. The complexity of the environment requires the agent associate complicated speech commands with high-fidelity visual observations, without floor plan map. The floor plans can be visualized and interacted with in https : //iithhot allenai rrg//emo/...
"Pense"
"Put the music"
"Tam on the lamp"
"Lights on"
"Lights off
"Bring shoes"
Figure 9: Visualization of the iTHOR environment with paired images and voices from the FSC dataset.
14

---

B.5
List of Tasks
Table 3: List tasks in each environment.
Envs
Tasks the original domain
Changes in the new domain
activate the floor lamp
unseen furniture, room decoration, room arrangement, and voices from new speakers
deactivate the floor lamp activate the TV deactivate the TV find the pillow
iTHOR
activate the light bulb deactivate the light blub mute the alarm clock pick up the pill case
unscen desk, object locations, object appearance, and sound from new speakers or the alarms
Desk
first block
unseen sound or voices
second block
Row
third block
forth block
first object
unseen camera intrinsics, camera extrinsics, background, relative locations among the objects, and voices from new speakers
second object
Row real
third object
forth object
C
Sound Data
Table 4:
Sound signals used in the experiments.
Dataset
Sound
Examples
activate light deactivate light
"Turn on the lights," "Lamp on" "Switch off the lamp," "[mp,""Loe "Lights off"
FSC
activate music
"Put on the music," "Play"
deactivate music bring shoes
"Pause music." " "Stop" "Get me my shoes," "Bring shoes"
"1," "3" names of objects
"zero," "one," "two," "three" "house" "tree," " "bird," "dog"
GSC
NSynth
C4. D4. E4, F4
Various instruments, tempo, and volume
US8K
bark, jackhammer
Sound recorded the wild
ESC-50
Clock alarm
Alarm sound emitted from various alarm clocks
bring pill case first object
"Pass over the pill box for me," "Give me the pill case" would like the first object." "Give me the leftmost object"
"Would you mind giving me the second object from the left," "Bring the third object from the right to me"
second object
Synthetic
"Take the third object," "Bring me the second object from the right"
third object
"Take the third object from the left"
fourth object
"Give the rightmost object to me "" "Hand over the fourth object"
15

---

D Visualization of task execution
D.1 Row
Sound: "On"" Target second bleck from the kill
"Zeo", Tagget Gom
DOLL
Sound: "On"",
Trrget: second block frem the lef
Figure 10: Visualization of the task execution in the Row environment after training without fine-tuning. The sounds come from Wordset dataset. Kuka moves its gripper the target block successfully in all episodes.
D.2 Desk
Sound:
"Give me the pill case", Torget pick the pill case
Sound: "Switch off the lights", Target: off the light bulb
Sound: "Lampp on", Target: turn on the light bulb
Sound: Alarm clock sound, Target. press the button mute the alumm
Figure 11: Visualization of the task execution in the Desk environment after training without fine-tuning.
16

---

D.3 iTHOR
Sound: "Bring me my shoes", Target: pick up the pillow
Sound: "Switch off' the lights", Target: turn off the floor lamp
Sound: "Play". Target turn the TV
Sound: "Stop the music", Target: tum the TV
Visualization of the task execution in the iTHOR environment after training without fine-tuning. Figure 12:
The sounds from FSC datasct. iTHOR agent finishes household tasks successfully all episodes.
17

---

D.4 iTHOR fine-tuning
Sound: "Stop the music".. Target: Turn off the TV
Before
After
Sound: "Fetch my shoes", Target: Pick up pillow
Before
Aller
Sound: "Tur the lamp", Target: Turn the floor lamp
Before
Afler
Sound: "Turn off the lamp", Target: Turn off the floor lamp
Before
Afler
Figure 13: Visualization task execution iTHOR environment before the fine-tuning unseen floor plans and the sound commands given by new speakers.
18

---

D.5 Desk fine-tuning
Sound: Alarm sound, Target: Mute the alarm by pressing the butlon
Before
After
Sound: "Bring me the pill case", 'Target: Pick up the pill case
Before
Aller
Sound: "Bring pill case", Target: up pill case
Before
Afler
Sound: "Turn off the lamp", Target: Turn the light bulb
Before
Afler
Figure 14: Visualization of execution Desk environment before and after the fine-tuning with unseen desk and the sound commands given by new speakers. The appearance of the desk and the pill case are different from the original desk. The location of the light bulb, the button, the LED, and the drawer are different from the original desk.
19

---

E Facts and details
E.1 Time efliciency
We evaluate the time efficiency of all the methods. All the models running single Nvidia GTX 1080 Ti GPU and a Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz. We report the average time in second (s) for the model to take one action in the iTHOR environment with the FSC dataset. The average is calculated from 12500 samples.
ANR: 0.041s
E2E: 0.018s
VAR: 0.024s
MAR::AR: 0.022s
E.2
ANR
Implementation: We first off-the-shell automatic speech recognition (ASR) named Mozilla DeepSpeech [53] to transcribe the speech to text. We then train learning- based natural language understanding (NLU) module to handle the noisy output from the ASR [54]. For example, "Play the music" is sometimes transcribed as "by the music." Finally, a ission-based RL agent operates with the predicted intent from the NLU.
Accuracy intent prediction of ASR+NLU: FSC dataset: 86.0%; Wordset: 87.0%.
Fine-tuning details: We fine-tune the RL agent of this pipeline because it is the major source performance degradation. The policy network initialized with the weights obtained during the training. During the fine-tuning, the input of the model includes images and ground-truth intent IDs as label of the images and reward signals from the simulator supervision signals. The policy network is updated based on the PPO loss and the auxiliary loss for the images.
ASR+NLU is perfect. The model also receives a one-hot
E.3 E2E
Fine-tuning details: We fine-tune the whole policy network end-to-end. The policy network is initialized with the weights obtained during the training. During the fine-tuning, the input of the model includes images and raw sound signals. The model also receives a one-hot label of the images, ground-truth intent IDs, and reward signals from the simulator as supervision signals. The policy network updated based the PPO loss and the auxiliary losses for the images and audio.
E.4 VAR
Fine-tuning details: We fine-tune the VAR and let the VAR provide rewards and observations to fine-tune the policy network. Both VAR and the policy network are initialized with the weights obtained during the training. We collect visual-audio triplets consisting of an image, positive audio, and negative audio from the environment and fine-tune the VAR using the triplet loss. During the fine-tuning of the policy network, the input includes images and vector embeddings from VAR. The policy network updated based on the PPO loss.
20

---

E.5 DiR
We show the statistics of training and fine-tuning data for Dif-VAR.
Table 5: Number of training and fine-tuning pairs
Envs
# of training pairs of fine-tuning pairs
Ratio (Fine-tuning training)
Row real
5.0 x 104
3.0 x 102
0.6%
Desk
2.5 10*
1.5 x 102
0.6%
iTHOR
6.0 10a
2.5 X 102
0.4%
We report the change in success rate w.r.t (1) label usage for fine-tuning Dif-VAR, and (2) number of RL steps given the fine-tuned representations with the specific number of labels.
Table Success rate with varying label usage and RI. fine-tuning steps for Ours(F) the Desk and iTHOR environments.
RL Steps
Envs
Episode Length
LU
0.1M
0.5M
1.0M
50
77.5
79.5
79.5
Desk
100
100
78.0
80.0
84.0
150
78.0
82.0
84.5
80
39.4
45.3
56.3
iTHOR
50
160
45.7
56.7
68.4
250
58.6
77.5
85.8
Table 7: Success rate with RL fine-tuning steps for Ours(F) in the Row-real environments.
RL Steps
Envs
Episode Length
LU
1100
2200
Row-real
20
300
40.0
77.5
21

---

E.6 Intermediate fine-tuning performance
0 lahels
labels
80
160 labels
250 labels
714
60.4
17.9
Success Rale (%)
47.0
18.8 LIL.
10.2
ANR
E2E
VAR
Ours (IF
Ours (H)
Methods
Figure 15: Success rate with varying number labels the iTHOR environment.
0 labels
50M1 90,1
50 labels
194.5
labels
150 labels
60
Success Rate (%)
300
44.5
44.5
44.5
15
40
VAR
Ours (F)
Ours (H)
Methods
Figure 16: Success rate with varying number of labels in the Desk environment.
22

---

Qualitative comparison of the representation
We visualize the VARs by projecting images and sounds to the joint space, as shown in Fig. 17. We see that the embeddings the same concept form cluster and clusters are separated from each other. Compared to VAR, the clusters in Mif- have better intra-cluster cohesion and inter-cluster separation, suggesting that the two distinct concepts better distinguished and the same concepts better related. During fine-tuning, although Dif- VAR does not have S as explicit indication of negatives like VAR does in the input, Bif- can still maintain relatively clear inter-cluster separation and provide reliable rewards for the self ieprovemtee of RL agents.
(0)
(b)
(d)
Figure Visualizations the VARs the ITHOR environments with FSC. The colors indicate the ground truth intent ID of embeddings of sound (marked by triangles) and image (marked by cincles). (a) VAR after the training. (b) VAR after the lune-nuning. (c) Dif- VAR after the training. (d) DiR-VAR after the fine-tuning.
G Illustration of data collection
labels (ANR, assuming classes total
DII-VAR
VAR
This image comesponds "Tam on the TV*
This
TTrm di TV*
This mage Tum or telight
"Tum TV" t0 0.
this s 0.0
Only providing ng posiive pairs afficeet (1 (shel) and int.livw
Providiig both pooittvo and negativa paits requi we more work 12 label)). and negadive pairs can be unintaltve
Providiing dioos indloos also raquiras more work (bell) and lees intuilive than providing pairs
Figure Comparison among different data collection methods. Left: Our method only asks pairing an image with an audio without the need for a class label. Middle: VAR requires an additional pairing process than our method. Right: ANR and E2E require two assignments of the underlying class label which is not intuitive and need more effort.
23


Reviews:
[{"review_id": "tongHHrXxe", "reviewer": "Reviewer_HeXC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper develops an approach for building command-following robots that can continually improve after deployment. The approach is based on image-audio representation learning that can be used to generate an intrinsic reward function for reinforcement learning on the robot. Experiments on simulated navigation and manipulation settings, and on real world robotic grasping, demonstrate that the approach can be used to continually improve robotic policies based on user commands, with less data compared to prior approaches.", "summary_of_recommendation": "I am not recommending accept because I have a few questions/concerns regarding the usefulness and feasibility of the approach for real world deployment.\n\n--- AFTER REBUTTAL ---\n\nI am increasing my score to weak accept after discusions because my concerns regarding more complex tasks than grasping are slightly orthogonal to the main goals of the paper - however, for the approach to be actually adopted by the community, I do believe these slightly more complex tasks are necessary to be shown as applicable for the proposed reward function.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please refer to the review above. \n\n- why is specifying commands in the form of sound is more novel or useful than specifying them in the corresponding language instructions?\n\n- how general is the approach in terms different sounds from diverse people with differences in tone, speed, accent etc. ?\n\n- the real world results mention an hour of fine-tuning for robotic grasping. Is the approach feasible for more complex / long horizon tasks beyond grasping that would potentially require a lot more time for fine-tuning with RL in the real world?\n\n- Some relevant comparisons are missing. How does the approach compare with prior works that learn vision-language embeddings? In my understanding the representation learning model can be replaced with a vision-language representation model and the rest of the pipeline including intrinsic reward computation can be kept as it is. \n\n- [minor] For sample-efficient deployment in the real-world, can the approach be combined with bootstrapping via few demonstrations? Since RL in the real-world in highly sample inefficient.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of Strengths\n\n- the framework of combining video and audio to obtain a joint representation space that can be used to define an intrinsic reward, in novel to the best of my knowledge. \n\n- the approach relies on very little domain knowledge about the environment/objects, and uses only monocular RGB image observations from an uncalibrated camera, combined with sound signals from humans.\n\n- the experiments are on diverse scenarios ranging from manipulation to navigation, and there are some results with real robots showing that the approach works for sim2real transfer in robotic grasping.\n\n- the paper is generally easy to follow and has sufficient details on the method for understanding.\n\nSummary of Weaknesses/Questions\n\n- intuitively, it is unclear to me why specifying commands in the form of sound is more novel or useful than specifying them in the corresponding language instructions (assuming sound to language mapping is near accurate, which is an implicit assumption made in the paper). \n\n- it is unclear how general is the approach in terms different sounds from diverse people - I did not see details about diversity of new speakers in terms of tone, speed, accent etc. \n\n- the real world results mention an hour of fine-tuning for robotic grasping. Is the approach feasible for more complex / long horizon tasks beyond grasping that would potentially require a lot more time for fine-tuning with RL in the real world?\n\n- Some relevant comparisons are missing. How does the approach compare with prior works that learn vision-language embeddings? In my understanding the representation learning model can be replaced with a vision-language representation model and the rest of the pipeline including intrinsic reward computation can be kept as it is. \n\n- [minor] For sample-efficient deployment in the real-world, can the approach be combined with bootstrapping via few demonstrations? Since RL in the real-world in highly sample inefficient.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "a27vqR2l5Ks", "reviewer": "Reviewer_b57y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method for contrastive learning on image and audio pairs for robotic tasks. There is an intrinsic reward function being modeled. The policy is trained in sim and then finetuned to real environments. The paper explores the use of audio conditioned robotics policies. ", "summary_of_recommendation": "The paper proposes a new framework for audio visual learning using contrastive self supervised loss. They've results on sim and real. The formulation is a little bit complicated and hard to understand so I think the paper could be written slightly better from a flow/explaining abbreviations perspective. The reward function + learning in sim makes the system not the most scalable, hence incremental advance. Overall the work is scientific and well executed. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. Why embed in audio space as against language space?\n2. How would you make the reward function more robust?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Reward function is described as a similarity between two vectors that are learned representations of visual-language, this does not factor in actual policy success, or success rate. I don't think this is a correct formulation of reward ie, how likely is it to succeed given state and action. There is no action component, language is only intent.\n\nThe contrastive loss however is an interesting formulation. It allows for fluidity in language descriptions, but presented in audio form.\n\nThe real world results are not very strong, but cleaner studies in sim do drive a point.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "RD4i99LHdP", "reviewer": "Reviewer_S2KB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel method for applying RL to robotics. The input of the system is audio command + visual. It has two steps. 1) using contrastive loss to train a audio+visual model to measure whether the audio and visual are close to each other. 2) using the audio+visual model to automatically calculate reward for RL training. The experimental results show improvements over existing baseline methods.", "summary_of_recommendation": "I recommend acceptance of this paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. For the real world fine tuning experiment, it will be great to see the performance can keep improving w/ RL exploration.\n2. This method should be compatible w/ any RL algorithm. It will be great to see how it works w/ another RL algorithm in addition to PPO.\n3. In theory, this method can also support language instead of audio. Would be interested in seeing how it can be combined w/ LLMs.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The overall idea is novel and the paper is well written. Reward design is a pain for RL and data efficiency is a huge problem. The method proposed in this paper can solve these two problems very well.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "LcNSkHPqE7", "reviewer": "Reviewer_TXeX", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method of using pairs of images and sounds (e.g. spoken words, phrases, and other sounds) as labels with which to train and fine tune models used for robotic command-following tasks. These sound-image pairs are used to train DiF-VAR, which embeds these sounds and images into a network that produces unit vectors in close proximity to the related sound or image. This trained DiF-VAR is then used to create a reward function used to train an RL agent on some robotic task. Following initial deployment, fine tuning can be performed to improve performance of both the DiF-VAR and the RL policy using only sounds as labels. This system is tested in simulated and real environments. The authors claim that this system provides substantial benefit to the deployment of consumer-facing learning robotic systems due to the relative ease of fine tuning in novel environments.", "summary_of_recommendation": "Overall, this paper is of fair quality and is clearly written. The system shown is meaningfully novel, but I'm not sure that the system is compelling enough for this system to have an impact on future works. If the authors are able to demonstrate their claim that this system is easy to fine tune for non-technical users compared to alternatives, then I would be more confident in the utility of this work going forward. They should also address the disparity in how they count the number of labels used by their own method versus others, which could cause these results to be overly optimistic.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. I’m not entirely clear on the benefit of using direct sound processing over translating sound into text first. As this is central to the novelty of this paper, I recommend expanding upon it.\n2. Under “Fine-tuning in novel domains”, I’d like to see more data that shows model improvement during fine tuning over fine tuning episodes collected, not just labels used. The label use information can and should be included here, but having both would provide a clearer picture.\n3. In the data given in Table 2, the number of labels used to demonstrate improvement in fine tuning seem to be arbitrarily selected- 253, 150, 300. Could you provide more clarification on why these are the label limits for each task? Perhaps intermediate results could be shown, e.g. finetuning improvement for each 100 additional labels?\n4. Your real-world fine tuning evaluation is based on a very low number of episodes- only 40 episodes total were used to evaluate your fine tuned model in the real world, if I'm understanding the wording on line 308 correctly. At that low number, the result could be skewed by more than +- 2% based on a single additional episode. Given the inherent unpredictability of learned robotic systems, that variance could add up quickly, for better or for worse. Would it be possible to perform more experiments for finer grained results?\n5.  The \"definition of labels\" given may be overly generous to the method described here, in which an audio-visual pair is described as using only a single label. Why are image and sound not counted as independent labels, given that they are two separate pieces of data? Could the same logic not be applied to other methods to combine multiple pieces of data into a single label? \n6. Small nit: In line 138, you refer to the latent space of the output of your indicator function as “joint space”. This is an overloaded term in robotics, as joint space could also refer to a vector whose components are the displacements of a robotic joint. You may want to consider changing that verbiage for the sake of clarity.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- It addresses the issue of continual learning in new environments, which is a relevant problem to the real-world deployment of learning robots.\n- It attempts to mitigate the cost of acquiring datasets for training and fine tuning, which is always a concern for learned robotics.\n- The proposed system is meaningfully novel.\n\nWeaknesses:\n- The paper does not adequately support its claim that this method of data labeling is easier or quicker than other methods.\n- The definition of labels given may be overly generous to the method described here, in which an audio-visual pair is described as a single label rather than two labels. If this is expressed as two labels rather than one, the results would be much different compared to other methods.\n- Real-world experimentation is limited to a simple grasping task.\n- This work is not relevant outside the realm of audio-commanded robotics, and within that realm it is limited enough in its application that I don't believe it is likely to see further use.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "9F7k39_nYO", "reviewer": "Reviewer_Tifq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a pipeline for training and fine-tuning visual-audio representations that can be exploited to generate intrinsic rewards for RL training on household voice-controlled manipulation tasks. In the representation learning stage, given (image, voice command audio, task_id) pairs, the authors use the contrastive loss to supervise the learning of visual and audio embedding, unifying the embeddings space from the two modalities, and pulling the embeddings for the same task id together. In the reinforcement learning stage, image observation and goal voice command audio are encoded and their cosine similarity is regarded as the intrinsic reward for RL training. When the robot is deployed in a new environment, by collecting (image, voice command audio) pairs and fine-tuning the representation with self-supervised contrastive loss, the model adapts to the new environment efficiently. Simulation and real-world results show the effectiveness of the method.", "summary_of_recommendation": "The reason for acceptance:\n\nthe authors propose to combine visual-audio representation learning and RL to train voice-controlled manipulation agent. The authors show that the proposed method outperforms baseline alternatives. In addition, the proposed method is easier and more effective to fine-tune and adapt to novel domains with data that can easily be collected by non-expert users. Solid simulation and real-world experiments are shown to support the main claim.\n\nThe reason for weak acceptance:\n\nThe pipeline is quite similar to the VAR paper except for some modifications in loss which means that the work is incremental in essence. The representation learning, RL, and fine-tuning pipeline are also quite standardized.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. How does the performance vary given different amounts of extra fine-tuning data? Show how efficient it is. 250 pairs are still a lot of data to collect for non-expert users. If you can plot a figure showing SR(y-axis) vs the amount of data (x-axis), it would provide more insights about how data-efficient the method is.\n2. When “unheard sound commands” is mentioned in the Experiment “Fine-tuning in novel domains” session, does it mean that the tasks are novel or the tasks are seen task but only the voice is unseen?\n3. It will be good if more information regarding the training data can be shared in the appendix. How many data pairs are needed for the pretraining? What is the ratio between pretraining and fine-tuning data amount?\n4. It will be good if more information regarding the list of tasks can be shared in the appendix. What tasks are the robot trained on? Are there novel tasks in evaluation?\n5. How are the baselines fine-tuned? Do they require task labels to fine-tune?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Quality\n\nVoice-controlled manipulation is a crucial and necessary competence a household robot should have and not a lot of work focus on the voice part. Most of the existing methods convert the voice command to a text command with a speech-to-text model and proceed with language models for downstream tasks. But the authors explore a more end-to-end way and propose a solid and clear pipeline to train the visual-audio representation, to train an RL agent with auto-generated rewards, and to fine-tune the representation in a new domain for further RL training. The method is clearly introduced. Simulation and real-world experiments are carried out to support and prove the effectiveness of the proposed method to some extent. \n\n### Clarity\n\nThe structure of the paper is clear and it is easy to read.\n\n### Originality\n\nThe representation learning part, the RL pipeline, and the fine-tuning part are quite standardized. The pipeline is also similar to VAR.\n\n### Strength\n\n1. Proven effectiveness of combining visual-audio representation learning with RL training in voice-controlled manipulation tasks.\n2. Proven effectiveness of fine-tuning with the proposed method.\n3. A solid solution to solve voice-controlled manipulation tasks.\n\n### Weakness\n1. The authors haven’t shown how the amount of fine-tuning data influences the results. The ratio between the pretraining data and fine-tuning data is also not shown. It is good to see that the fine-tuning works, but there still lacks experiments showing how “data-efficient” the proposed method is regarding data amount. \n2. More details regarding the experiments are appreciated like the list of evaluated tasks, what tasks are novel, and how many data pairs are collected for pretraining. More details regarding how the baselines are fine-tuned are appreciated.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for reward learning, which specifically gives a reward function that is (i) conditioned on a user audio command, and (ii) also a function of the current environment image.  In this sense, the reward function is a visual-audio one, where the audio specifies the task, and the image provides the current environment observation, which together determine the reward.  The reward learning method is combined with RL to learn policies, and tested on a few different simulated environments, and in a type of multi-object reaching task in the real world.

The reviewers appreciated various aspects about the paper, including the overall goal of having audio-specified goals, and that "reward design is a pain for RL and data efficiency is a huge problem." (S2KB). There were comments from multiple reviewers that perhaps just running state-of-the-art speech-to-text, and/or involving LLMs, could be suitable, but in the rebuttal the authors I believe have argued convincingly that using audio processing that doesn't go through text has several beneficial aspects, including being able to capture non-linguistic sounds, and potentially having less intermediate errors.  Some reviewers appreciated this more "end-to-end" way of doing audio-specified rewards.  Aside from performance implications, it is also interesting and merits study.

There are some weaknesses as well that were pointed to, some of which were addressed in rebuttal responses.  In general, although I personally can appreciate some of the challenges of the real-world example (uncalibrated camera, no known-object assumptions), the demo does come off as looking simplistic, in part because it seems essentially a 1-degree-of-freedom task (I know it's "2", but it's basically 1) with no contact.  Still, the reviewers thought the sim evaluations "drove a point" (b57y) and the real-world results were better than having no real-world results.  The work does seem incremental with respect to VAR, which is not necessarily a bad thing, and the authors clarified that there are a couple practical benefits of DifVAR including not needing triplet-paired data.

Overall, the authors engaged in good discussion with the reviewers.  One reviewer (TXeX) increased their score to Weak Accept after robust responses by the authors to their questions.  Another reviewer (Tifq) also increased their score to Weak Accept after discussion with reviewers and the AC after which they decided their concerns were orthogonal to the main point of the paper.  Another reviewer (Tifq) had multiple rounds of questions, and were satisfied by the authors' responses.

Also, it should be noted that most roboticists do not typically use audio signals, and especially not typically for task specification in a way that doesn't first get converted to text.  I think this is a plus for the paper, that it addresses a type of problem that is not that common.

Overall, I think the paper will be of sufficient net benefit to the robot learning community to recommend its acceptance.  Please include the updated plots on sample efficiency, and the comparison with the Whisper-based method, in the final camera ready version.
Decision: Accept (Poster)