Noisy Agents: Self-supervised Exploration by Predicting Auditory Events
Authors: Chuang Gan
Xiaoyu Chen
Phillip Isola
Antonio Torralba
Joshua B. Tenenbaum
Venue:  (2021)
Keywords: Audio Curiosity
RL exploration

Abstract: Humans integrate multiple sensory modalities (e.g., visual and audio) to build a causal understanding of the physical world. In this work, we propose a novel type of intrinsic motivation for Reinforcement Learning (RL) that encourages the agent to understand the causal effect of its actions through auditory event prediction. First, we allow the agent to collect a small amount of acoustic data and use K-means to discover underlying auditory event clusters. We then train a neural network to predict the auditory events and use the prediction errors as intrinsic rewards to guide RL exploration.  We first conduct an in-depth analysis of our module using a set of Atari games. We then apply our model to audio-visual exploration using the Habitat simulator and active learning using the TDW simulator. Experimental results demonstrate the advantages of using audio signals over vision-based models as intrinsic rewards to guide RL explorations.

Under review as a conference paper at ICLR 2021
NOISY AGENTS: SELF-SUPERVISED EXPLORATION BY PREDICTING AUDITORY EVENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Humans integrate multiple sensory modalities (e.g., visual and audio) to build a causal understanding of the physical world. In this work, we propose a novel type of intrinsic motivation for Reinforcement I Learning (RL) that encourages the agent to understand the causal effect of its actions through auditory event prediction. First, we allow the agent to collect a small amount of acoustic data and use K-means to discover underlying auditory event clusters. We then train a neural network to predict the auditory events and use the prediction errors intrinsic rewards to guide RL exploration. We first conduct an in-depth analysis of our module using set of Atari games. We then apply model to audio-visual exploration using the Habitat simulator and active learning using the ThreeDWorld (TDW) simulator. Experimental results demonstrate the advantages of using audio signals over ausion-basee models as intrinsic rewards to guide RL explorations.
INTRODUCTION
Deep Reinforcement Learning algorithms aim to learn a policy of an agent to maximize its cumulative rewards by interacting with environments and have demonstrated substantial success in wide range of application domains, such as video game (Mnih et al., 2015), board games (Silver et al., 2016), and visual navigation (Zhu et al., 2017). While these results are remarkable, one of the critical constraints is the prerequisite of carefully engineered dense reward signals, which are not always accessible. To overcome these constraints, researchers have proposed a range of intrinsic reward functions. For example, suriosity-driven intrinsic reward based on prediction error of current (Burda et al., 2018b) or future state (Pathak et al., 2017) on the latent feature spaces have shown promising results. Nevertheless, visual state prediction is non-trivial problem as visual state is high-dimensional and tends to be highly stochastic in real-world environments.
The occurrence of physical events (e.g. objects coming into contact with each other, or changing state) often correlates with both visual and auditory signals. Both sensory modalities should thus offer useful cues to agents learning how act in the world. Indeed, classic experiments in cognitive and developmental psychology show that humans naturally attend to both visual and auditory cues, and their temporal coincidence, to arrive at rich understanding of physical events and human activity such as speech (Spelke, 1976; McGurk & MacDonald, 1976). In artificial intelligence, however, much more attention has been paid the ways visual signals (e.g., patterns in pixels) can drive learning. We believe this misses important structure learners could exploit. As compared to visual cues, sounds are often more directly or easily observable causal effects of actions and interactions. This is clearly true when agents interact: most communication uses speech or other nonverbal but audible signals. However,
just as much physics. Almost any time two objects collide, rub or slide against each other, or touch in any way, they make sound. That sound is often clearly distinct from background auditory textures, localized in both time and spectral properties, hence relatively easy to detect and identify; in contrast, specific visual events can be much harder separate from all the ways high-dimensional pixel inputs are changing over the course of a scene. The sounds that result from object interactions also allow to estimate underlying causally relevant variables, such as material properties (e.g., whether objects are hard or soft, solid or hollow, smooth, or rough), which can be critical for planning actions.
These facts bring a natural question of how to use audio signals to benefit policy learning in RL. In this paper, our main idea is to use sound prediction an intrinsic reward to guide RI. exploration.

---

Under review as a conference paper at ICLR 2021
Intuitively, we want exploit the fact that sounds are frequently made when objects interact, or other causally significant events occur, like cues to causal structure or candidate subgoals an agent could discover and aim for. A naive strategy would be to directly regress feature embeddings of audio clips and use feature prediction errors as intrinsic rewards. However, prediction errors on feature space do not accurately reflect how well an agent understands the underlying causal structure of events and goals. It also remains an open problem on how to perform appropriate normalizations to solve intrinsic reward diminishing issues. To bypass these limitations, we formulate the sound-prediction task as a classification problem, in which we train neural network to predict auditory events that occurred after applying action to a visual scene. We use classification errors as an exploration bonus for deep reinforcement learning. Concretely, our pipeline consists of two exploration phases. In the beginning, the agent receives an incentive to actively collect a small amount of auditory data by interacting with the environment. Then we cluster the sound data into auditory events using K-means. In the second phase, we train a neural network to predict the auditory events conditioned on the embedding of visual observations and actions. The state that has the wrong prediction is rewarded and encouraged to be visited more. We demonstrate the effectiveness of our intrinsic module on game playing with in Atari (Bellemare al., 2013), audio-visual exploration in Habitat (Savva et al., 2019), active learning using a rolling robot in ThreeDWorld (TDW) (Gan et al., 2020a). In summary, our work makes the following contributions:
We introduce a novel and effective auditory event prediction (AEP) framework to make use of the auditory signals as intrinsic rewards for RL exploration.
We perform an in-depth study on Atari games to understand our audio-driven exploration works well under what circumstances.
We demonstrate our new model can enable a more efficient exploration strategy for audio-visual embodied navigation on the Habitat environment.
We show that our new intrinsic module is more stable in the 3D multi-modal physical world environment and can encourage interest actions that involved physical interactions.
RELATED WORK
Audio-Visual Learning. In recent years, audio-visual learning has been studied extensively. By leveraging andio-visual correspondences in videos, can help learn powerful audio and visual representations through self-supervised learning (Owens et al., 20160: Aytar et al., 2016; Arandjelovic & Zisserman, 2017; Korbar al., 2018; Owens & Efros, 2018). In contrast to the widely used correspondences between these two modalities, we take step further by considering sound as causal effects of actions.
The problem of exploration in Reinforcement Learning (RL) has been active research topic for decades. There are various solutions that have been investigated for encouraging the agent to explore novel states, including rewarding information gain (Little & Sommer, 2013), surprise (Schmidhuber, 1991; 2010), state visitation counts (Tang et al., 2017; Bellemare et al.,
RL Explorations.
2016), empowerment (Klyubin et al., 2005), curiosity (Pathak et al., 2017; Burda et al., 2018a) disagreement (Pathak et al., 2019) and so on. A separate line of work (Osband et al., 2019; 2016) adopts parameter noises and Thompson sampling heuristics for exploration. For example, Osband et al. (2019) trains multiple value functions and makes use of the bootstraps for deep exploration. Here, we mainly focus on the problem of using intrinsic rewards to drive explorations. The most widely used intrinsic motivation could be roughly divided into two families. The first one is count- based approaches (Strehl & Littman, 2008: Bellemare et al., 2016; Tang et al., 2017; Ostrovski et al., 2017; Martin et al., 2017; Burda et al., 2018b), which encourage the agent to visit novel states. For example, Burda et al. (2018h) employs the prediction errors of self-state feature extracted from a fixed and random initialized network as exploration bonuses and encourage the agent to visit more previous unseen states. Another one is the curiosity-based approach (Stadie et al., 2015; Pathak et al., 2017; Haber et al., 2018; Burda et al., 2018a), which is formulated as the uncertainty in predicting the consequences of the agent's 's actions. For instance, Pathak et al. (2017) and Burda et al. (2) use the errors of predicting the next state in the latent feature space as rewards. The agent is then encouraged to improve its knowledge about the environment dynamics. In contrast to previous work purely using vision, make use of the sound signals as rewards for RI. explorations.
2

---

Under review as a conference paper at ICLR 2021
Stage Sound Clustering
sound clusters
online clustering
(sound,)
sound,
(sound(+1)
class F+1
sounde+1
Intrbusic
Loss
Reward
y (image;;
class
imager
action
Stage 2: Auditory Event Prediction
Figure 1: An overview of our framework. Our model consists of two stages: sound clustering and auditory event prediction. The agent starts collect diverse set of sound through limited environment interactions (i.e. 10K) and then clusters them into auditory event classes In the second stage, the agent uses errors of auditory events predictions as intrinsic reward to explore the environment.
Sounds and Actions. There are numerous works to explore the associations between sounds and actions. For example, Owens et al. (2016a) made the first attempt collect audio-video dataset through physical interaction with objects and train an RNN model to generate sounds for silent videos. Shlizerman al. (2018); Ginosar et al. (2019) explore the problem of predicting body dynamics from music and body gesture from speech. Gan et al. (2020b) and Chen et al. (2020) introduce an interesting audio-visual embodied task in 3D simulation environments. More recently, Dhiraj et al. (2020) collected a large sound-action-vision dataset using Tilt-bolt and demonstrates sound signals could provide valuable information for find-grained object recognition, inverse model learning, and forward dynamics model prediction. More related to us are the papers from Aytar et al. (2018) and Omidshafiei et al. (2018), which have shown that the sound signals could provide useful supervisions for imitation learning and reinforcement learning in Atari games. Concurrent to our work, Dean et al. (2020) uses novel associations of audio and visual signals as intrinsic rewards to guide RL exploration. Different from them, we use auditory event predictions intrinsic rewards to drive RL explorations.
METHOD
In this section, we first introduce some background knowledge of reinforcement learning and intrinsic rewards. And then elaborate on the pipeline of self-supervised exploration through auditory event predictions. The pipeline of our system is outlined in Figure 1.
3.1 BACKGROUND
MDPs We formalize the decision procedure in context standard Markov Decision Process
(MDP), defined as (S, A,, 7). S. A and pus) S + [0. 1] denote the sets state, action and the distribution of initial state respectively. The transition function T(''ls, a) S A S [0, 1] defines the transition probability to next-step state s' if the agent takes action a at current state 8. The agent will receive reward after taking action according to the reward function R(s, a). discounted by E (0, 1). The goal of training reinforcement learning is to learn an optimal policy T² that can maximize the expected rewards under the discount factor ?
Σ R(s, t)
π arg max EçET
(1)
3

---

Under review as a conference paper at ICLR 2021
a) Victory
b) Landing
Falling
a) Victory
b) Blow - up
c) Overheat Warning
Figure 2:
The first and second show the auditory events that discovered by the K-means algorithm in Frostbite and Assault respectively.
where ( represents the agent's trajectory, namely {(s0. ag), (51, ...,...... The agent chooses an action from a policy (a|s) S x A + [0, 1] that specifies the probability of taking action a E A under state s. In this paper, we concentrate on the MDPs whose states are raw mmage-baser observations well audio clips, actions discrete, and T is provided by the game engine.
Intrinsic Rewards for Exploration. Designing intrinsic rewards for exploration has been widely used resolve the sparse reward issues in the deep RI communities. One effective approach is to use the errors of a predictive model as exploration bonuses (Pathak et al., 2017; Haber et al., 2018 8; Burda et al., 201 2018a). The intrinsic rewards will encourage the agent to explore those states with less familiarity. We aim to train a policy that can maximize the errors of auditory event predictions.
3.2 REPRESENTATIONS OF AUDITORY EVENTS
Consider an agent that sees a visual observation Su,t, takes an action a, and transits the next state with visual observation s.,++1 and sound effect $,,++1 The main objective of our intrinsic module to predict auditory events of the next state, given feature representations of the current visual observation 8,, and the agent's action az We hypothesize that the agents, through this process, could learn the underlying causal structure of the physical world and use that to make predictions about what will happen next, and as well as plan actions to achieve their goals.
To better capture the statistic of the raw auditory data, we extract sound textures (McDermott Simoncelli, 2011) ((8..t) to represent each audio clip 8s,t. There are many possible ways compute audio features. We choose sound texture mainly for its computation efficiency and effectiveness in audio-visual learning (Owens al., 20:6); For the task of auditory event predictions, perhaps the most straightforward option is to directly regress the sound features ($$,.t+1 given the feature embeddings of the image observation Su,t and agent's actions 0t- Nevertheless, we find that not very effective. We hypothesize that the main reasons are: 1) the mean squared error (MSE) loss used for regression is satisfied with "blurry" predictions. This might not capture the full distribution over possible sounds and a categorical distribution over clusters; 2) the MSE loss does not accurately reflect how well agent understands these auditory events. Therefore, we choose instead define explicit auditory events categories and formulate this auditory event prediction problem as a classification task, similar (Owens al., 2016b).
3.3 AUDITORY EVENTS PREDICTION BASED INTRINSIC REWARD
Our AEP framework consists of two stages: sound clustering and auditory event prediction. We need to collect a small set of diverse auditory data in the first stage and use them to define the underlying auditory event classes. To achieve this goal, we first train an RL policy that rewards the agents based on sound novelty, And then, we run a K-means algorithm to group these data into fixed number of auditory events. In the second phase, we train forward dynamics network that takes input as the embedding of visual observation and action and predicts which auditory event will happen next. The prediction eиror is then utilized events with more uncertainty, thus improving its ability to understand the consequences of its actions. We will elaborate the details of these two phases below.
intrinsic reward to encourage the agent to explore those auditory
4

---

Under review as a conference paper at ICLR 2021
Sound clustering. The agents start collect audio data by interacting with the environment. The goal of this phase is to gather diverse data that could be used to define auditory events. For this purpose, we train an RL policy by maximizing the occurrences of novel sound effects. In particular, we design an online clustering- based intrinsic motivation module to guide explorations. Assuming we have a series of sound embeddings ((8,,)), f 1,2,... 2,... T and temporarily grouped into K clusters. Given a new coming sound embedding (sa,e+1), we compute its distance to the closest cluster centers and use that as an exploration bonus. Formally,
r1 min (10(s..+11 c:l12
where rt denotes an intrinsic reward at time t, and C;, E {1, }} represents a cluster center. During this exploration, the number of clusters will grow, and each cluster's center will also be updated. Through this process, the agent is encouraged to collect novel auditory data that could enrich cluster diversity. After the number of the clusters is saturated, clustering algorithm (Lloyd, 1982) on the collected data to define the auditory event classes and use the center of each cluster for the subsequent auditory event prediction task. To be noted, the number of cluster K is determined automatically in our experiments. In practice, we define K' E [5, 30) for clustering at each time step and use the silhouette score to automatically decide the best K, which measure of how similar a sound embedding to its own cluster compared to other clusters. When the best K keeps unchanged for updates, we believe it is "saturated". We visualize the corresponding visual states in two games (Frostbite and Assault) that belong to the same sound clusters, and it can be observed that each cluster always contains identical or similar auditory events (see Figure 2).
then perform the K-Means
Auditory event predictions. Since have already explicitly defined the auditory event categories, the prediction problem can then be easily formulated as a classification task. We label each sound tex- ture with the index of the closest center, and then train forward dynamics network f ((V (s.,,), ay; 0p) that takes the embeddings of visual observation V (8a,s) and action a as input to predict which auditory event cluster the incurred sound ((5..(+1) belongs to. The forward dynamics model is trained on collected data using gradient descent to minimize the cross-entropy loss L between the estimated class probabilities with the ground truth distributions Wt-1 as:
L Loss( (S az; 0p)-yt+1)
The prediction is expected to fail for novel associations of visual and audio data. We use the loss L as intrinsic reward. will reward the agent at that stage and encourage it to visit more such states since it is uncertain about this scenario. In practice, we do find that the agent can learn avoid dying scenarios in the games since that gives similar sound effect it has already encountered many times and can predict very well. By avoiding potential dangers and seeking novel events, agents can learn causal knowledge of the world for planing their actions to achieve the goal.
4 EXPERIMENTS
As proof of concept, we first conduct experiments on Atari game environments (Bellemare et al., 2013) to understand our model. And then we apply our model to more realistic 3D environments, such Habitat (Savva et al., 2019) and TDW (Gan et al., 2020a).
4.1
SETUP
Atari Game Environment Our primary goal is to investigate whether we could use auditory event prediction as intrinsic rewards to help RL exploration. For this purpose, we use the Gym Retro (Nichol et al., 2018) 8) as testbed to measure agents' competency quantitatively. Gym Retro consists of diverse set of Atari games, and also supports an audio API to provide the sound effects of each state. We use 20 familiar video games that contain sound effects to compare the intrinsic reward only exploration against several previous state-of-the-an intrinsic modules. We follow the standard setting in (Pathak et al., 2017; Burda al., 201 8b), where agent use external rewards an evaluation metric to quantify the performance of the exploration strategy. This is because doing well on extrinsic rewards usually requires having explored thoroughly.
Audio- Visual Explorations on Habitat Following (Dean et al., 2020), we use the the apartment in Replica scene (Straub et al., 2019) with the Habitat simulator for experiments. This scene has
5

---

Under review as a conference paper at ICLR 2021
Table 1: Category results of 20 Atari games according to the dominant type of sound effects. We label the games in which our method performs the best in bold font.
Dominant sound effects
Atari games
Amidar, Carnival, NameTThisamme Frostbite, FishingDerby, MsPacman AirRaid, Assault, Jamesboud. ChopperCommane StarGunner,
Event-driven sounds
Action-driven sounds
Tutankham, WizardOrWor, Gopher, DemonAttack Asteroids, Freeway, TimePilot, Battleoone, CrazyClimber
Background sounds
211 unique locations as the state space, and the action space contains three discrete actions, namely move forward, turn right and turn left. We follow the setting from (Chen et al., 2020) by placing one fixed audio source at fixed location. The audio engine in Habitat could simulate various sounds at arbitrary agent receiver positions based on the room's s geometry, major structures, and materials, so that the agent can hear different sounds when moving. The agent is trained without extrinsic reward, and we compare the steps needed to fully explore the scene.
Active learning with rolling robot TDW We finally test module on 3D physic simulation platform, built on top of TDW (Gan et al., 2020a). As shown in Figure 7 we place rolling robot agent (i.e,, red sphere) on billiard table. The agent is required execute actions to interact with objects of different materials and shapes. When two objects collide, the environment could generate collision sound based on the physical properties of objects. We would like compare the agent's 's behaviors in this 3D world physical environment using different intrinsic rewards.
Baselines. We consider four state-of-the-art intrinsic motivation modules for comparisons, including Intrinsic Curiosity Module (ICM) (Pathak et al., 2017), Random Feature Networks (RFN) (Burda al., 201 8a), Random Network Distillation (RND) (Burda al., 201 8a). and Model Disagreement (DIS) (Pathak et al., 2019). We use their open-source code to reproduce the results.
Implementation details. For all the experiments, we choose PPO algorithm (Schulman et al., 2017) based on the PyTorch implementation to train our RL agent since it is robust and requires little hyper-parameter tuning. We use the open- toolbox experiments on Atari games, we use gray-scale image observations of size 84x x4 and 60ms audio clip. We set the skip frame $-4 for all the experiments. We use a 4-layer CNN as the encoder the policy network. As for the auditory prediction network, choose 3-layer CNN with output channel 32, 64, 64 to encode the image observation and use 2-layer MLP with 512 hidden units to predict the auditory events. For all experiments, is also included the beginning of each reported curve. The intrinsic reward in both phases are clipped to [-1, after normalization. We use 1, and 8 parallel environments for Atari, Habitat, and TDW experiments, respectively.
extract sound texture features. For
model use 10K interactions for stage exploration, which
4.2 EXPERIMENTS ON ATARI
Overall Results. Figure 3 summarizes the evaluation curves of mean extrinsic reward in 20 Atari games. For each method, we experiment with three different random seeds and report the standard deviation in the shaded region. As the figure shows, our module achieves significantly better results than previous vision-only intrinsic motivation modules in 15 out of 20 games. Another interesting observation is that the carned score keeps going up. even only using intrinsic rewards.
Result Analysis. We would like to provide an in-depth understanding of under what circumstances algorithm works well. The sound effects in Atari games fall into three different categories: 1) event-driven sounds which emitted when agents achieve a specific condition (e.g., picking up a coin, the explosion of an aircraft, etc.); 2) action-driven sounds which emitted when agents implement a specific action (e.g., shooting, jumping, etc.) and 3) background noise/music. According to the dominant sound effects in each game, we summarize the 20 Atari games in Table 1. It is also worth noticing that, the sounds in Atari games include positive (e.g. carning a bonus), negative (e.g. dying) and meaningless sound (e.g. background music). None of the category accounts for the majority.
1htpps:/ ://i thub. hoom//nddewwwwnss... mult 1t i sensory/blob//aeeeeeeeeeeeeeeee subband.

---

Under review as a conference paper at ICLR 2021
AirRaid
Amidar
Assault
Carnival
500
300
Jamesbond
MsPacman
NameThisGame
CrazyClimber
133
Loco0
599
FishingDerby
StarGunner
Tutankham
WizardorWor
2000
130
500
ChopperCommand
Asteroids
Freeway
BattleZone
5000
Gopher
Frostbite
TimePilot
DemonAttack
ICM
RND
RFN
Ours
Figure 3: Average extrinsic rewards of our model against baselines in 20 Atari games. Intrinsic rewards used for training, and the extrinsic rewards used for evaluation only.
Based on the category defined in Table and the performance shown in Figure 3, we can draw three conclusions. First, both event-driven and action-driven sounds boost the performance of our algorithm. Since the sound is more observable effects of action and events, understanding these casual effects essential learn better exploration policy. Second, our algorithm performs better on the games dominant with event-driven sounds compared to those with action-driven sounds. We believe that event-driven sounds contain higher-level information, such as the explosion an aircraft collecting coins, which can be more beneficial for the agent to understand the physical world. Third, our algorithm falls short in comparison with baselines when the sound effects mainly consist of meaningless background noise or background music (i.e. CrazyClimber, MsPacman, Gopher, Tutankham, WizardOfWor, and BattleZone). These are also reasonable since our noisy agents are incentivized make interventions that lead to making surprising sounds. Sometimes sound events will occur independently of the agent's 's decisions and do not differentiate between different policies, and therefore do not guide learning. The events that guide learning are those that are a consequence. directly or indirectly, of an agent's s actions. The agent is encouraged to understand the causal effect of its actions and thereby take actions that lead to better exploration.
Ablated Study. We further carry out additional experiments to study the following questions of model design.
Predict auditory events or sound features? One main contribution of our paper is to use auditory event prediction as an intrinsic reward. We conduct an ablated study by replacing this module with sound feature prediction module. We train neural network that takes the embedding of visual state and action as input and predicts the sound textures. The comparison curves are plotted in Figure 4. We observe that the auditory event prediction module indeed earned more rewards. We speculate that auditory events provide more structured knowledge of the world, thus lead to better RL explorations.
Sound clustering or auditory event prediction? We adopt two-stage exploration strategy. A natural question is if this is necessary. We show the curve using phase-1 only (i.e. sound clustering) in Figure 4. This is novelty driven exploration strategy. The reward function is defined as the
7

---

Under review as a conference paper at ICLR 2021
Amidar
BattleZone
Carnival
Amidar
BattleZone
Carnival
icea
1403
8
6o00
500C
630
steas
million staps DV!
rid
Predict Sound
Clustering Only
Ours
Random
RND
Active
Comparisons on earned extrinsic re- Figure 5: Comparisons earned extrinsic wards between our auditory prediction mod- wards using random, RND, and our active explo- ule and sound feature prediction module.
Figure 4:
ration strategy respectively.
distance to the closest clusters. We notice that the returned extrinsic reward is similar to sound feature prediction, but worse than auditory event prediction.
Is active exploration necessary? We propose online fustering-based intrinsic module for active audio data collections. To verify its efficacy, we replace this module with random explorations and RND (Burda et al., 20) For fair comparisons, we allow both models to use 10K interaction data to define the event classes with the same K-mean clustering algorithm. The comparison results are shown in Figure We can find that the proposed active explorations indeed achieve better results. We also compute the cluster distances of these models and find that the sound clusters discovered by active exploration are more diverse, thus facilitating the agents to perform in-depth explorations.
4.3 EXPERIMENTS ON HABITAT
To evaluate the exploration ability of agents trained with approach, report the unigue state coverage, given a limited exploration budget. Figure 6, using only 16K exploration steps, agent has already explored all unique states (21 states), while other methods have visited less than 195 unique states (190, 157, 170, and 195 for ICM, RND, RFN, and DIS, respectively) on average, Even exploring for 20k steps, all other methods have not yet fully explored the 211 states. These results demonstrate that our agent explores environments more quickly and fully, showing the potential ability of exploring the real world. We believe the reason that our auditory event prediction encourages agents to explore the novel areas with uncertain sounds.
Visiee 1
DI5 Durs
Figure 6: Comparisons of stage coverage on Habitat.
4.4 EXPERIMENTS ON TDW
Collision
Intrinsic Rewards
1.0
0.8
0.6
0.4
150 steps (K)
Interaction steps (K)
ICM
RND
RFN
DIS
Ours
Figure 7: Explorations on a multi-moda. physics environment. From left right: physical scene, collision events, and intrinsic reward changes
Setup. We take an image observation of 84 size and 50ms audio clip as input. We use three- layer convolutional network to encode the image and extract sound textures from the audio clip. Same as the previous experiment, of moving to eight directions and stop. An action is repeated times on each frame. We run all the experiments for 200K steps with 8 parallel environments.
train the policy using the PPO algorithm. The action space consists
8

---

Under review as a conference paper at ICLR 2021
Result Analysis. To understand and quantify the agent's behaviors in this 3D physical world, show the number of collision events and intrinsic rewards in Figure 7. We noticed two major issues with the previous rision-based curiosity models. First, the prediction errors the latent feature space could not accurately reflect subtle object state changes in the 3D photo-realistic world, in which physical event happens. Second, the intrinsic reward can diminish quickly during training, since the learned predictive model usually converges to a stable state representation of the environment. Instead, our auditory event prediction driven exploration will lead agents interact more with objects in the physical world, which is critical to learning the dynamics of the environments.
4.5 DISCUSSIONS
This section aims to examine our module leverages the knowledge of different auditory events for RL explorations.
Do we need to perform event classifications? We aim to understand the role that the auditory event prediction module plays in our learning framework. There are two alternatives to leverage multi- sensory signals. One option to design a simple binary classification task by predicting making a sound or not(See Bin Cls in Figure 8). The other one is to predict the associations between visual frames and audio clips (Dean et al., 2020) (See SHE in Figure 8). We will reward an agent its prediction is incorrect. Otherwise, it will be reward. We conduct these ablated studies in 3 Atari games, Habitat, and TDW environments. From Figure 8, we observe that predicting auditory events can indeed lead better exploration strategy. These results further demonstrate that understanding the causal structures of different events is essential for multi-sensory RI explorations.
Amidar
BattleZone
Carnival
TDW
Habitat
Extrinac ba.
steps 04)
million on slepe (81)
tat steps P4)
Inceraction steps 10
Staps (K)
Img Cls
Bin Cls
SHE
Figure 8: Ablated study on 3 Atari games, Habitat and TDW.
Do we need audio to define events? We also want to understand if it is necessary to use audios to define high-level events. Perhaps, the results will also be good if we replace the audio features with visual features to cluster the event types. We plot the curves of clustering visual frame to get the event type (see Img Cls in Figure 8). The results are significantly worse than using audio features. We hypothesize that it might be hard to capture the patterns of pixel changes when these events happen, but sounds are more observable to discover these events.
CONCLUSIONS AND FUTURE WORK
In this work, introduce intrinsic reward function of predicting sound for RL exploration. Our model employs the errors of auditory event prediction as an exploration bonus, which allows RL agents explore novel physical interactions of objects. We demonstrate our proposed methodology and compared it against number of baselines on Atari games. Based on the experimental results above, we conelude that sound conveys rich information and is powerful for agents to build a causal model of the physical world for efficient RL explorations. We hope our work could inspire more works on using modai-yo cues for planing and control. In the future, we plan to explore more effective audio representations and combine vision-driven approaches to improve RL. exploration results further.
REFERENCES
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 609-617. IEFR, 2017. 2

---

Under review as a conference paper at ICLR 2021
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances in Neural Information Processing Systems, pp. 892-900, 2016.
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtbbe. In NIPS, pp. 2930-2941, 2018.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In NIPS, pp. 1471-1479, 2016. 2
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013. 2,5
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study curiosity-driven learning. ICLR, 2018a. 2, 4, 6, 8, 14
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. ICLR, 2018b. 1, 2,5
Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Kr- ishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments, 2020. 3,
Victoria Dean, Shubham Tulsiani, and Abhinav Gupta. See, hear, explore: Curiosity via audio-visual association. arXiv preprint arXiv:2007.03666 2020. 3, 5,
Gandhi Dhiraj, Gupta Abhinav, and Pinto Lerrel. Swoosh! rattle! thump! actions that sound. 2020.
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threedworld: A platform for interactive multi-moda. physical simulation. arXiv preprint arXiv:2007. 2004020077.4454, 2020a. 2, 5,
Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. ICRA, 2020b. 3
Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learning individual styles of conversational gesture. In CVPR, pp. 3497-3506, 2019. 3
Nick Haber, Damian Mrowca, Stephanie Wang, Li F Fei-Fei, and Daniel L Yamins. Learning to play with intrinsicall--motiaaeed. self-aware agents. NIPS, pp.
2018. 2,
Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: A universal agent-centric measure control. In 2005 IEEE Congress on Evolutionary Computation, volume 1, pp. 128 125-13,,20 2005.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Co-training of audio and video representations from self-supervised temporal synchronization. arXiv preprint arXiv: 1XXi:1807.002300 2018. 2
Daniel Ying-Jeh Little and Friedrich Tobias Sommer. Learning and exploration in dation-perception loops. Frontiers neural circuits. 7:37, 2013.
Stuart P. Lloyd. Least squares quantization in pem. IEEE Trans. Inf. Theory, 28:129-136, 1982. 5
Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Mareus Hutter. Count-based exploration in feature space for reinforcement learning. IJCAI, 2017. 2
Josh H McDermott and Eero P Simoncelli. Sound texture perception via statistics of the auditory periphery: evidence from sound synthesis. Neuron, 71(5):926-94) 2011.
Harry McGurk and John MacDonald. Hearing lips and seeing voices. Nature, 264(5588):746, 1976.
10

---

Under review as a conference paper at ICLR 2021
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-553 533, 2015.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprinl arXiv: arXiv:/804. 1788040037000 2018. 5
Shayegan Omidshafiei, Dong-Ko Ki Kim, Jason Pazis, and Jonathan How. Crossmodal attentive skill learner. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 139-146. International Foundation for Autonomous Agents and Multiagent Systems, 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. bootstrapped dqn. In NIPS, pp. 4026-4034, 2016. 2
Deep exploration via
Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized value functions, 2019. 2
Georg Ostrovski, Mare G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In JCML, pp. 2721-2730, 2017. 2
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. ECCV, 2018. 2
Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. In CVPR, pp. 2405 2413, 2016a. 3,
Andrew Owens, Jiajun Wu, Josh H McDermott, William T' Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. European Conference on Computer Vision, pp. 801-816. Springer, 2016b. 2,
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-drive -driven exploration by self-supervised prediction. In JCML, 2017. 1, 2, 4, 5,
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. ICML, 2019.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, al. Habitat: A platform for embodied ai research. In ICCV, pp. 1339-93479 2019. 2,
Jürgen Schmidhuber. possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-22222222222 1991.
Jürgen Schmidhuber. Formal theory creativity, fun, and intrinsic motivation (1990-2010), IEEE Transactions on Autonomous Mental Development, 233)-23, 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17070063777 2017, 6
Eli Shlizerman, Lucio Dery, Hayden Schoen, and Ira Sanellmache-SShizzemmee Audio to body dynamics. In CVPR, pp. 7574-7583, 2018. 3
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieserr, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Elizabeth Spelke, Infants' intermodal perception of events. Cognitive psychology, 8(4):553-560, 1976.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration reinforcement learning with deep predictive models. arXiv preprint IrXiv:1577.000 2015. 2
11

---

Under review as a conference paper at ICLR 2021
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: digital replica of indoor spaces. arXiv preprint WrXiv:190600579, 2019.
Alexander L Strehl and Michael Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences,
2008.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. In NIPS, pp. 2753-2762 2017.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA, pp- 3357-3364, 2017. 1
12

---

Under review as a conference paper at ICLR 2021
A
TRAINING DETAILS
Table 2 shows the hyper- hyper-paramene parameters used in our algorithm.
Table 2: Hyper-parameters used in our algorithms.
Hyperparameter
Value
Rollout length
128
Number of minibatches
Learning rate
2.5e-4
Clip parameter
0.1
Entropy coefficient
0.01
X
0.95
7
0.99
B
NAVE BASELINE FOR MULTI-MODAL RL
AirRaid
Amidar
Carnival
Frostbite
Derbing
Auaroge Satreas Nawerrss
mmn
115
130
M
100
Caps
millon Steps
(90
million steps
coeps
Figure 9: Average extrinsic rewards of our model against baselines combined with sound in 5 Atari games
In this section, we carry out ablated experiments to demonstrate that the gains in our method are caused by the audio-event prediction, rather than the use of multi-modality information. For four baselines (i.e. RND, RFN, ICM, and DIS), instead of predicting audio-event, they consider sound information by concatenating both visual and sound features to predict the image embedding in the next time step. As shown in Figure 9, our algorithm significantly outperforms other baselines in five Atari games. This indicates that it is non-trivial to exploit sound information for RL, and our algorithm benefits from the carefully designed audio-event prediction as intrinsic reward.
C HARD EXPL.ORATIONS GAMES
In this section, investigate whether the audio-driven intrinsic rewards could be further utilized to improve policy learning in the hard exploration scenarios, where extrinsic rewards exist but very sparse. We use six hard exploration environments in Atari games, including Venture. Solaris, Private Eye. Pitfall!, Gravitar and Montezuma's 's Revenge for experiments. Following the strategy proposed
PrivateEye
Pitfall
MontezumaReveng
Venture
Frames (N)
PPO
RND
DIS
ICM
RFN
Our
Figure 10: Comparison of combining intrinsic and extrinsic reward on 6 hard exploration Atari Games.
13

---

Under review as a conference paper at ICLR 2021
by RND (Burda et al., 2018a), we two value heads separately for the intrinsic and extrinsic reward module and then combine their returns. We also normalize intrinsic rewards to make up the variances among different environments. We compare
model against those vision-only baselines including ICM, RND, RFN, DIS mentioned in the main paper. All experiments are run for 4 million steps with 32 parallel environments. We use the max episodic returns to measure the ability of explorations.
The comparison results are shown in Figure 10. Note the X-axis here refers to the frames rather than parameter updates like RND. All the baselines are reproduced using their open-source code. Our approach achieves best/comparable results in 4 out of 6 Atari games.
14


Reviews:
[{"review_id": "TP1-tbQv-8Y", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript proposes a novel method on how to include audio signals as rewards for image-based reinforcement learning (RL) exploration. More specifically, the agent collects audio signals, identifies the individual audio sources and trains a classifier to forecast the audio cues. The forecasting error in the next time step is used as reward for RL. This encourages the behaviour to explore novel audio cues. Authors’ apply the method to different set of problems (Atari games, Habitat simulator and active learning TDW simulator) and find that the proposed method improves the agent’s behaviour/performance.\n\nCombining audio and video information is essential to improve the explorative behaviour of agents that explore multisensory environments, especially if the relationships between agent behaviour and sounds are causal. In this case, the audio signals provide useful information much earlier in comparison to time than images could. Hence, the proposed approach is relevant for improving the explorative behaviour of agents in multisensory environments. \n\nThe paper is well structured, and the description of the methods is clear. The description of the experiments is not so clear. Note, that the reviewer is not familiar with these environments. For example, in the Habitat experiment, how many sound sources have been used? One in each room? If additional sources were strategically placed in different rooms to favour the proposed method, then one can expect that performance increases. Also, the reviewer found it challenging to interpret the results. For example, Figures 3 shows that the reward of the proposed methods is higher than the reward of other methods. It is not clear, however, how much the performance of the different methods is better than chance level performance. High reward means better performance, however, was the agent able to win game levels? Would be helpful to learn more about how “good” the agent was able to perform the task. Or to clarify which reward level was needed for the agent to successfully explore the environment. Moreover, when assessing the random exploration performance, how was random exploration performed? Each action was randomly selected based on a uniform distribution. If so, then this may not be the most reliable way to estimate random exploration.\n\nOverall, the paper proposes an interesting approach that enable agents to autonomously explore an environment based on unknown auditory cues. \n\nThe pros:\n-\tIntegration of multisensory information in multisensory environments is essential when the information provided by the different sensors is independent and hence contributes to development of more successful behaviour. \n-\tThe idea to use the prediction error to encourage explorative behaviour is smart\n-\tMethod evaluated in on different environments\n\nThe cons: \n-\tInterpretation of performance is not straight forward\n-\tA more systematic analysis of the relationship between relevant sound sources and environment sounds would be helpful to get an idea about the potential and constraints of the proposed method \n\n\n### Update after discussion period ###\nGood idea, but the results don't clearly support the authors claims. I lowered my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Meg_dkWWJCP", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper investigates the incorporation auditory events into reinforcement learning. Specifically, it proposes a new algorithm that uses event prediction as an intrinsic reward. This algorithm has two phases. In the first phase, an agent is given a small number of episodes to gather diverse auditory data. This phase has two pieces that work in conjunction: 1) As it learns, the agent clusters the sound embeddings it has observed using K-means. 2) To encourage the agent to find diverse auditory data, it is rewarded for reaching states that emit sounds that are far away from its existing cluster centers. In the second  phase (which dominates the first in terms of number of episodes consumed), a new agent is trained to predict the cluster center to which the next auditory event will belong. It is rewarded according to how incorrect its prediction is, thereby encouraging the agent to explore states for which it has difficulty predicting the auditory event.\n\nThe paper performs experiments in Atari, Habitat and TDW. It compares against strong vision-based intrinsic reward baselines. It also performs experiments comparing its proposed methodology to ablations with the aim of determining 1) whether it suffices to predict sound features instead of auditory events, 2) whether a two-stage exploration strategy is necessary, 3) whether it is necessary to perform active exploration in the first phase, and 4) whether event classification is necessary.\n\n# Writing\n\nThis submission does not read as that of a paper ready for publication. Its organization, unnecessary use of the passive voice, singular-plural inconsistencies, tense mixing, and muddled descriptions weaken its value. Below is a (non-exhaustive!) list of issues.\n\n- Abstract\n\n“We first conduct an in-depth” \n\nThere needs to be a transition from the method description for this “first” to fit here.\n\n- Introduction\n\nWhy is Deep Reinforcement Learning all caps?\n\n“algorithms aim to learn a policy of an agent to maximize its cumulative rewards by interacting with environments”\n\nsingular/plural\n\n“domains, such as video game”\n\nGame should be plural\n\n“While these results are remarkable, one of the critical constraints is the prerequisite of carefully engineered dense reward signals, which are not always accessible.”\n\nIs Go a good example of this? AlphaZero accomplishes the same task without carefully engineered dense reward signals.\n\n“For example, curiosity-driven intrinsic reward based on prediction error of current (Burda et al., 2018b) or future state (Pathak et al., 2017) on the latent feature spaces have shown promising results.”\n\nThis sentence is ordered awkwardly. As written, “have” refers to “curiosity-driven intrinsic reward”, which is singular. “on the latent feature spaces” doesn’t work here.\n\n“visual state is high-dimensional” -> states are\n\n“speech or other nonverbal but audible signals” -> speech or other audible signals OR speech or nonverbal, audible signals\n\nUsing “other” here makes it read as if speech is a member of “nonverbal but audible signals”\n\n“However, it is just as much in physics.”\n\n ?\n\n“A naive strategy would be” -> A naive strategy is\n\n“In the beginning”\n\nDoes this mean in the first phase?\n\n“The state that has the wrong prediction is rewarded and encouraged to be visited more.”\n\nPassive voice makes this hard to parse. The agent is the one making predictions and receiving rewards.\n\n“understand our audio-driven exploration works well under what circumstances”\n\nSome words are missing here\n\n“can encourage interest action that involved physical interaction”\n\nNeeds fixing\n\n- Related Work\n\n“By leveraging audio-visual correspondences in videos, it can help to learn powerful audio and visual representations through self-supervised learning”\n\nIt can help to learn? -> It can learn?\n\n“Reinforcement Learning (RL)”\n\nAcronym has already been introduced.\n\n“makes use of the bootstraps for deep exploration”\n\nbootstraps -> bootstrap\n\n“Here, we mainly focus on the problem of using intrinsic rewards to drive explorations.”\n\nWhy is exploration plural?\n\n“The most widely used intrinsic motivation could be roughly divided into two families.” could -> can\n\nThe works discussed in the ensuing sentences are already listed and cited in the previous sentences. If you are going to discuss as if you had not already just mentioned them, it is better to exclude them from the previous sentences.\nAlso, maybe you should add “approaches” or “methods” somewhere in this sentence.\n\n“Burda et al. (2018b) employs the prediction errors of a self-state feature extracted from a fixed and random initialized network and encourage the agent to visit more previous unseen states.” previous -> previously\n\nWhat is a self-state feature? Burda does not use that term and there is no explanation here.\nThe end of the sentence doesn’t make sense. What is the subject of “encourage”?\n\n“Another one is the curiosity-based approach (Stadie et al., 2015; Pathak et al., 2017; Haber et al., 2018; Burda et al., 2018a), which is formulated as the uncertainty in predicting the consequences of the agent’s actions.”\n\n-The paper previously said that there were two families so “another one” should be “The other” or something else signifying that this is referring back to the two families comment.\n\n-The paper described family one as a set of approaches (plural) but family two as an approach “singular”.\n\n-The end of the sentence needs fixing.\n\n“The agent is then encouraged to improve its knowledge about the environment dynamics” then -> thereby\n\nFigure 1 would be more clear if it did not use time indexing for the stage 1 section.\n\n“There are numerous works to explore”\n\nthat explore?\n\n“More recently, Dhiraj et al. (2020) collected a large sound-action-vision dataset using Tilt-bolt and demonstrates sound signals could provide valuable information for find-grained object recognition”\n\n-Dhiraj et al. is plural so demonstrates should be demonstrate.\n\n-could provide -> can provide\n\n-find-grained -> fine-grained\n\n“More related to us” us -> our work OR this work\n\n“which have shown that the sound signals could provide useful supervisions for imitation learning and reinforcement learning in Atari games.” \n\n-could -> can\n\n-unnecessary passive voice\n\n\nThroughout the Sounds and Actions paragraph, the paper repeatedly switches between describing papers in past tense and present tense.\n\n“And then we elaborate on the pipeline of self-supervised exploration through auditory event predictions.” And then -> Then\n\nThere is nothing wrong (in general) with starting a sentence with the word “And” but this is not an appropriate place for it.\n\n“a standard Markov Decision Process (MDP), defined as (S, A, r, T , µ, γ). S, A and µ(s) : S → [0, 1] denote”\n\nas -> by\n\nIt is bad form to start a sentence with a symbol.\n\n“The transition function T (s 0 |s, a) : S × A × S → [0, 1] defines the transition probability to next-step state s’ if the agent takes action a at current state s.”\n\nThe transition function defines this transition probability whether or not the agent executes a at s.\n\n“The goal of training reinforcement learning is to learn an optimal policy π ∗ that can maximize the expected rewards under the discount factor γ as”\n\nThis is the goal of reinforcement learning not the goal of training reinforcement learning.\n\nThe sentence doesn’t work. It reads “The goal is to learn an optimal policy that is an optimal policy.” Either modify to “The goal is to learn an optimal that is an optimal policy.” or “The goal is to learn an optimal policy. An optimal policy is …” \n\n“The agent chooses an action a from a policy π(a|s)” from -> according to\n\n“Intrinsic Rewards for Exploration”\n\n This paragraph is repeating information that was already written\n\n“Designing intrinsic rewards for exploration has been widely used to resolve the sparse reward issues in the deep RL communities”\n\n-Unnecessary passive voice\n\n-What deep RL communities? Isn’t there just one? If there are more than one, what are they?\n\n“transits to the next state with visual observation sv,t+1 and sound effect ss,t+1. “\n\ntransits -> transitions\n\nstate with -> state, receiving\n\n“We hypothesize that the agents, through this process, could learn the underlying causal structure of the physical world and use that to make predictions about what will happen next, and as well as plan actions to achieve their goals.”\n\n-“and as well as” is redundant\n\n“To better capture the statistic of the raw auditory data”\n\nWhat statistic?\n\n“For the task of auditory event predictions, perhaps the most straightforward option is to directly regress the sound features Φ(ss,t+1) given the feature embeddings of the image observation sv,t and agent’s actions at”\n\nThe paper said this already\n\n“Nevertheless, we find that not very effective. We hypothesize that the main reasons are: 1) the mean squared error (MSE) loss used for regression is satisfied with “blurry” predictions. This might not capture the full distribution over possible sounds and a categorical distribution over clusters; 2) the MSE loss does not accurately reflect how well an agent understands these auditory events. Therefore, we choose instead to define explicit auditory events categories and formulate this auditory event prediction problem as a classification task, similar to (Owens et al., 2016b).”\n\n-Nevertheless is not appropriate here.\n\n-The paper runs this experiment and discusses the results later in the paper. Its confusing to discuss it both as a choice and as an ablation.\n\n\n“Our AEP framework”\n\nWhile the acronym can be deduced from the section header, it is still good practice to write it explicitly.\n\n“We need to”\n\nThis is a design choice, not a need.\n\n“And then”\n\nAnd is not appropriate here.\n\n“takes input as the embedding of visual observation and action and predicts which auditory event will happen next.”\n\nneeds fixing\n\n“then utilized” -> used\n\n“to explore those auditory events with more uncertainty”\n\nthose is unnecessary here\n\n“We will elaborate on the details of these two phases below.”\n\nGet rid of “will”\n\n“The agents start to collect audio data by interacting with the environment”\n\nThey start to isn’t good phrasing here. \n\n“During this exploration, the number of clusters will grow”\n\nfix tense\n\n“After the number of the clusters is saturated”\n\nThe description of the sound clustering process is muddled. It does not defined saturated until after it has used it in context.\n\n“To be noted, the number of cluster K is determined automatically in our experiments. In practice, we define K ∈ [5, 30] for clustering at each time step and use the silhouette score to automatically decide the best K, which is a measure of how similar a sound embedding to its own cluster compared to other clusters.”\n\nDo “To be noted” or “In practice” have any added value here?\n\n“we believe it is “saturated””\n\nThe paper is defining saturation, it is not a belief.\n\n“We visualize the corresponding visual states in two games (Frostbite and Assault) that belong to the same sound clusters, and it can be observed that each cluster always contains identical or similar auditory events”\n\n-Unnecessary passive voice\n\n-Always is a pretty strong claim. Can we deduce that from this visualization?\n\n“Since we have already explicitly defined the auditory event categories, the prediction problem can then be easily formulated as a classification task”\n\n-Don’t need “already”\n\n-Don’t need “then”\n\n“It will reward the agent at that stage and encourage it to visit more such states since it is uncertain about this scenario”\n\n-Use present tense tense\n\n-at that state?\n\n“In practice, we do find that the agent can learn to avoid dying scenarios in the games since that gives a similar sound effect it has already encountered many times and can predict very well.”\n\n-we do find -> we find\n\n-dying scenarios?\n\n- Experiments\n\n“And then”\n\nAnd is not appropriate here\n\n“Our primary goal is to investigate whether we could use auditory event prediction as intrinsic rewards to help RL exploration.”\n\nThis statement is repeated many times in the paper and has no added value to this section.\n\n“also supports an audio API to provide”\n\nto provide -> that provides\n\n“We use 20 familiar video games”\n\nIs familiar needed here?\n\n“We follow the standard setting in (Pathak et al., 2017; Burda et al., 2018b), where an agent can use external rewards as an evaluation metric to quantify the performance of the exploration strategy. This is because doing well on extrinsic rewards usually requires having explored thoroughly.”\n\n-Passive voice is excessive in multiple places here\n\n-incorrect use of interrogatives\n\n“the the” typo\n\n“Table 1: Categorical results”\n\nThese are not results\n\n“could simulate”\n\ncan simulate\n\n“physic simulation platform” physics\n\n“Figure 7 ,”comma misplaced\n\n“The agent is required to execute actions to interact with objects of different materials and shapes.”\n\nThe agent interacts with objects of different materials and shapes.\n\n“When two objects collide, the environment could generate collision sound based on the physical properties of objects”\n\ncould generate collision sound?\n\n“We would like to compare”\n\nWe compare\n\n“we choose PPO algorithm”\n\neither choose the PPO algorithm or choose PPO\n\n“The PyTorch implementation”\n\nwhich?\n\n“the open-source toolbox 1”\n\n-which?\n\n-footnote shouldn’t have space\n\n“As for the auditory prediction network”\n\nDon’t need “as”\n\n“our model use 10K interactions” fix\n\n“previous vision-only intrinsic motivation modules”\n\ndon’t need previous here\n\n“We would like to provide an in-depth understanding of under what circumstances our algorithm works well.”\n\nYou would like to or you do?\n\n“event-driven sounds which emitted when agents” fix\n\n“action-driven sounds which emitted when agents” fix\n\n“None of the category accounts for the majority” fix\n\n“Since the sound is more observable effects of action” fix\n\n“games dominant with event-driven” fix\n\n“These are also reasonable”\n\nNothing for “These” to refer to\n\n“Sometimes sound events will occur independently of the agent’s decisions and do not differentiate between different policies”\n\ndifferentiate is not the right word here\n\nThe phrase “ablated study” is used repeatedly. Use “ablation study” instead?\n\n“We further carry out additional experiments”\n\nfurther unnecessary\n\n“One main contribution of our paper is to use auditory event prediction as an intrinsic reward.”\n\nRepeated without added value\n\n“In Figure 6, using only 16K exploration steps, our agent has already explored all unique states (211 states)” fix\n\n“These results demonstrate that our agent explores environments more quickly and fully, showing the potential ability of exploring the real world.”\n\nDo they? This is a strong claim\n\n“less than 195 unique states” fewer\n\n“These results demonstrate that our agent explores environments more quickly and fully, showing the potential ability of exploring the real world.”\n\nDo they? Again, a strong claim.\n\nThere is a sub-header “Setup” of Section 4.4 Experiments on TDW. There is also a TDW sub-header of Section 4.1 Setup.\n\n“The action space consists of moving to eight directions and stop. An action is repeated 4 times on each frame.”fix\n\n“previous vision-based modules”\n\ndon’t need previous\n\n“could not” -> did not\n\n“the 3D photo-realistic world, in which a physical event happens.”needs fix\n\n“Instead, our auditory event prediction driven exploration will lead agents”\n\nIn constrast, … exploration leads agents\n\n“(See SHE in Figure 8)” see\n\n“We will reward an agent” We reward an agent\n\n“We also want to understand if it is necessary to use audios” audios?\n\n“is powerful for agents to build a causal model of the physical world” fix\n\n- Comments on Organization\n\n-I think it would make more sense to group the question-based analysis together. IE, the ablation studies and the discussions.\n\n-I think whether to use clustering classification or feature prediction should be discussed consistently throughout the paper. As it currently stands, sometimes it is presented as a design choice, sometimes with the claim that the latter is worse (without supporting evidence), and sometimes as a question to be answered by an ablation study.\n\n- Other comments\n\n“As compared to visual cues, sounds are often more directly or easily observable causal effects of actions and interactions.”\n\nIs this obvious?\n\n# Causality?\n\nThe paper makes a number of comments about its method learning causal structure. To me, these seem like big claims. The proposed algorithm has no mechanism that tests counterfactuals or, as far as I can tell, any other mechanism for estimating causation, so I see no reason why it would learn anything beyond correlative relationships. Given this fact, in my opinion, if the paper wants to make claims about the fact that its method is learning causal structure, it should back these claims up with experiments.\n\n# Experimental Evaluation\n\nThe paper makes very definitive claims (see writing section) about the effectiveness of its method compared to the baselines. In my opinion, there are a number of issues with these claims. First, the paper gives no information (that I could find) about how it tuned the baselines or whether they tuned them at all. Second, for the Atari results, the paper states that it used three random seeds. For the other experiments, it does not say how many seeds it used (or at least I could not find where it said so). Both of these facts make it difficult to know how seriously to take the claims of superior performance.\n\n\n# Related work\n\nThe paper cites many references in its related work section. Yet, I feel that it tells us almost nothing about what it is most important for us to know about:\n\n\"More related to us are the papers from Aytar et al. (2018) and Omidshafiei et al. (2018), which have shown that the sound signals could provide useful supervisions for imitation learning and reinforcement learning in Atari games. Concurrent to our work, Dean et al. (2020) uses novel associations of audio and visual signals as intrinsic rewards to guide RL exploration. Different from them, we use auditory event predictions as intrinsic rewards to drive RL explorations.\"\n\nWe are only given a couple of sentences of information about these papers. Additionally, I am not sure that is consistent that the paper claims that Dean is concurrent, but at the same time, designed its experiments to follow Dean “Following (Dean et al., 2020), we use the the apartment 0 in Replica scene (Straub et al., 2019) with the Habitat simulator for experiments.”\n\n# Choice of baselines\n\nThe paper appears to be concerned with claiming superior performance over vision-based intrinsic methods, yet it is not clear to me that having superior performance is necessary for the proposed method to have added value, given that the two methods make use of disjointed information for prediction targets. I do not mean to say that these experiments do not have added value—certainly it is nice to see the proposed algorithm compared to existing algorithms—but maybe the adversarial narrative is not the right choice? I understand that the paper is claiming that sounds give the agent better information for intrinsic exploration but, in my opinion, convincing evidence for that claim would require extensive experiments on a wide array of intrinsic methods using both sound and vision and many environments.\n\n# Broader Scope Question\n\nCan a paradigm requiring an initial exploration stage to collect diverse sounds be effective in environments in which some sound events require a large amount learning to discover?\n\n# Closing Thoughts\n\nThis seems like promising work, but in my opinion it is not ready for publication. Both the quality of writing and the issues with seeds/tuning independently merit rejection. There are also other issues discussed above.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CLSFVGl-Xk5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a novel type of intrinsic rewards for RL agents based on audio events prediction. A two stage method is proposed that includes sound clustering and auditory event prediction. The auditory event prediction error is used as an intrinsic reward for better exploration of the agents. The idea is evaluated on Atari games, and two 3D simulators: Habitat and TDW.\n\nStrengths:\n1. Decent idea to use audio as curiosity to drive the exploration of an agent.  \n2. Well motivated by how humans integrate multisensory inputs to understand the physical world.\n\nWeakness:\n1. Justification of causal effect of its actions. See detailed comments.\n2. Why not using bottom up clustering methods such as Agglomerative Clustering so that the number of clusters is not needed to be specified? The current method mannually set K to be in the range of 5 - 30, and then decide the best K is somewhat cubersome.\n3. Most of the results are using Atari games as proof of concept. The results on more realistic environements are not very convincing. What are the sound events in HABITAT? If an agent just navigates in the environment listening to the same sound, how it's going to help exporation at all? There is no event defined.\n4. The task is set up as an event classification task to get the intrinsic reward, so audio is actually not directly used. Technically, audio might not even be necessary in this case. What if directly using the meta data from the graphical engine to know the sound type or event type and use the ground-truth to guide the agent's learning process? This can serve as an upper bound for the proposed method to cluster audio events. Or, what if clusering the visual frames to get the event type? These are all useful baselines to demonstrate the use of audio is essential.\n\nDetailed Comments/Questions:\n1. The paper repetitively mention the agent is encouraged to understand the causal effect of its actions. However, the sound in experiments are usually just accompanying a visual event. Is it just audio-visual correspondence that helps or causality?\n2. Comparison to Dean et al. 2020 should be more clear. It is mentioned this work mainly studied if the sound signals along could be utilized as intrinsic rewards, which is confusing. What is the difference compared to Dean et al. 2020 is still not clear.\n3. In figure 4, how to directly use sound clustering as an intrinsic reward is not clear to this reviewer.\n4. An informative baseline would be to remove clusters, but set up the task as a binary classification task: making sound or not making sound. It would be informative to know whether the agent is really leveraging the knowledge of different sound events for exploration.\n5. Plenty of typos and grammar mistakes should be fixed.\n    - P2, RL Explorations, A separate line of work studies adopt?\n    - Sec. 3.2, to represent each audio clip s_t s_s,t ...?\n    - Sec. 4.1, how well an exploration stragegy is ;  We follows -> We follow; State-of-the-arts\n    ........\n\nJustification of recommendation:\nThis paper proposes a nice idea, but the paper is not very well written. The experiments needs further clarifications. This reviewer is happy to raise the score if these concerns can be addressed in authors' response.\n\n\n\n\n###Final Recommendation###\n\nBased on the discussions with other reviewers and AC, this paper is not ready to publish at this stage mainy due to the following reasons:\n1. the big claim of causality as also pointed out by R6\n2. the writing should be significantly improved and the experiments lack details as pointed out by all reviewers\n3. the new problems found during discussion with the AC regarding the ablation study, and seeds, etc. \n\nIn summary, this paper presents an interesting idea, but the experiments and writing in its current shape make the paper insufficient to be published at ICLR. The authors are encouraged to polish the paper in writing and experiments for future resubmissions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BFqOXf20bSX", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces an approach for using auditory event prediction to drive exploration, specifically by using the prediction error of auditory events as a shaping reward which is used along with the extrinsic reward. The approach consists of two phases. In the first phase, the agent explores the environment in a self-supervised manner and collects audio data, which are then clustered. The index of these clusters are the auditory events used during the second phase. In the second phase, an RL agent learns to predict the label of an auditory event at the next time step from the current observation and an action. The prediction error produced from this classification task is given as a shaping reward along with the extrinsic reward to the learning agent. The demonstrated results seem to suggest that this approach can improve over other self-supervised exploration methods in Deep RL.\n\nPros:\nOverall, the approach of using auditory events to drive exploration is novel.\nThe paper is well-written and presents the idea clearly.\n\nCons: \nSome implementation details about the approach are not available.\nThe approach seems to require a pre-training phase that requires collecting a significant amount of data.\nDoes not provide a clear intuition for why auditory events lead to better performance than other self-supervised approaches.\n\nQuestions:\nI have a few questions related to the approach, experiment results, and implementation details, which would help clarify my understanding of the paper.\n1. How are the intrinsic and extrinsic rewards combined to train an RL agent? The paper doesn’t seem to have any details about this. I assume it is the sum of extrinsic reward and a scaled-down intrinsic reward? If yes, then how was the scaling factor for the intrinsic reward chosen? And how was this chosen for the baseline methods?\n2. The approach seems to require a pre-training phase to identify different auditory events through K-means clustering. And this pre-training phase relies on generating behavior to seek out novel clusters. Would it be possible to avoid such a pre-training phase, and instead identify the auditory events as the agent learns to solve the main task? Would be interesting to see if the performance still holds in this scenario.\n3. In the results, the amount of data that was used for pre-training doesn’t seem to be included. It seems unfair to the other methods when the introduced method has experienced more data because of the pre-training phase. A fairer comparison would be to train the baseline agents longer to account for the pre-training phase and then make a comparison with the introduced method?\n4. In many of the Atari games, the baseline approaches seem to fail in learning? Could the authors comment on why this is the case? \n5. The main experiments (Fig. 3) on Atari seem to use 8 parallel environments. The choice of the number of environments would affect the batch size of the learning update and usually, it is common to use 16 or 32. Why was this choice made?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7oBpvD0PA9N", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update**: Other reviewers have pointed out issues with this paper's ablation study. Additionally, it is difficult to trust the empirical results because they are based on only three runs. In light of these criticisms, I have updated my score from a 7 to a 5. I still think this idea is neat and am generally a proponent of introducing audio into work on RL, but the experiments as presented in this submission do not currently paint a complete picture.\n\n**Summary**: This paper investigates sound perception as a means of intrinsic reward for reinforcement learning agents. Specifically, the proposed method rewards the agent for discovering state/action pairs which lead to sound events that are difficult to predict. Especially in environments where sound effects are correlated with high-level events, this strategy tends to improve performance over sound-agnostic baselines.\n\nOverall, I think this is an interesting paper. Audio events provide important queues which help humans understand the natural world and influence their decision. Such rich acoustic structure is often imitated in simulated environments, even in early Atari games. While substantial effort has gone into extracting as much usable information as possible out of the pixels of Atari games, the easily-accessible sound systems in these environments have been mostly ignored. It stands to reason that, especially for particular games, this information could be helpful.\n\nHypothesizing that encountering novel audio events may be informative for learning good policies, the authors propose a strategy which explicitly rewards agents for finding such states. They show that this strategy leads to improved performance on several environments, especially ones where audio information is associated with high-level events and interactions (e.g. balls colliding) as opposed to background noise (e.g. music in games).\n\nFrom an audio perspective (my primary area of expertise), the proposed strategy seems reasonable. The authors cluster perceptually-informed embeddings of audio slices to identify discrete classes, which makes sense when targeting environments whose audio primarily consists of sound effects (e.g. a clacking sound) synchronized to high-level events (e.g. two balls colliding). Why did the authors choose to use a texture-based sound embedding (McDermott & Simoncelli, 2011) as opposed to something more standard like log-amplitude Mel spectrograms? I would be quite interested to see how the performance of the latter compares to texture-based embeddings.\n\nThe experiments also seem reasonable: they compare the performance of the same policy-learning model/algorithm with different intrinsic reward sources on a slew of sparse-reward environments. The results seem to indicate that the proposed intrinsic reward usually leads to the best policy among the examined sources of intrinsic reward. One criticism is that it would be nice to see if these different intrinsic reward strategies are symbiotic; i.e., if multiple sources can be combined to improve performance. It would also have provided stronger evidence in favor of the proposed intrinsic reward function to see results on different policy-learning algorithms (besides PPO and CNNs). A caveat here is that I have limited experience on RL and defer to the expertise of the other reviewers in assessing the relevance of the baselines (ICM, RND, RFN, DIS).\n\nThe authors need to do more to distinguish their work from that of Omidshafiei et al. 2018. In a cursory overview of that work, it appears that there is a lot of overlap with the proposed work. It seems that the primary distinction is that the proposed work uses audio _only as part of the intrinsic reward_ (leaving the policy model unaware of audio cues), while the prior work _adds audio processing to the agent_. Can the authors comment further on the distinctions between their work and this prior work? Also, it's not immediately clear from the paper if the policy model (agent) receives audio as part of its state input; can the authors clarify?", "extra_scores": {}, "extra_text": {}}, {"review_id": "300cU_7wxuc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis work proposes the use of sound prediction as intrinsic reward to guide reinforcement learning (RL) exploration. Sounds are often directly related to causal effects of actions and interactions. By modeling different sound event classes via a clustering algorithm, the cluster prediction errors are used as intrinsic rewards for RL. The proposed approach is tested on three different simulation environments. The results on 22 different environments (20 Atari games, Habitat and TDW environments) illustrate that using exploration capability of the proposed approach and improved performance in comparison to other baseline methods leveraging intrinsic reward modules for RL exploration.\n\n########################\n\nPros:\n- The idea is novel and of interest to the RL community at large. The approach is clear and easy to follow. \n- Comprehensive experimental analysis and convincing results. Specifically, the analysis of dominant sound types in Atari games is well done. It helps understand that the Atari games for which there are no gains with the proposed method have dominantly background sounds. It also helps isolate the games in which sounds are causal effects of actions or events.\n- The ablative analysis demonstrates well that sound event prediction is a beneficial way to use audio as intrinsic rewards.\n- Testing on different variety of test beds also highlights the exploration capability of the proposed work in comparison to other intrinsic reward methods.\n\n########################\n\nCons:\n- The design choice of using texture features for audio is unclear and not motivated well. What properties do they capture that are desirable for the RL domains used for experimentation? It is also unclear how these are computed. Some more insights about the choice of audio features will be helpful to the reader.\n- If distances between sound texture features do not capture causal structure of the auditory events, then why are they suitable to be used for clustering? Wouldn’t the same problem carry over? Why should clusters be assumed to capture inherent causal structure when distances between sound texture features are used to create the clusters in the first place?\n- The writing requires more clarity (suggestions below).\n\n########################\n\nReason for score:\nThe idea is novel and results presented look great. The writing needs some rework (suggestions for improvements below) along with some explanations for feature computation and experimental conditions. I recommend for acceptance provided the authors revise the manuscript as per requested changes.\n---\nUPDATE: \nDue to concerns raised by other reviewers, and my own confusion about the computation of audio features, and clarifications on the causality stance, I have lowered my score from 8 to 7. \n\n########################\n\nQuestions during rebuttal:\n- Please refer to the questions in the Cons section and other feedback.\n- Comparisons are performed with methods using intrinsic motivation modules. How would the proposed work compare with other prior work where audio is used to aid RL (not necessarily as an intrinsic module), such as Aytar et al., 2018; Omidshafiei et al., 2018?\n\n\n########################\n\nSome typos and other feedback:\n- Consider using the \\citet command when referring to the authors of a reference paper in a sentence. (Eg: “Silver et al., 2016 concluded that…” versus “(Silver et al., 2016) concluded that …”)\n- Introduction, paragraph 1: “…a range of intrinsic reward function.” -> “…a range of intrinsic reward functions.”\n- Introduction, last paragraph (prior to bullets): provide citations for the Atari domain, and expand the first usage of TDW.\n- Related Work, RL explorations: Tompson sampling -> Thompson sampling\n- Related Work, RL explorations: Osband reference missing the year in the references section and the citation.\n- Related Work, RL explorations: “…uses the errors of predicting the next state…” -> “use the errors of predicting the next state…”\n- Figure 1 caption: “The agent start to collect a diverse set…” -> “The agent starts to collect a diverse set..”\n- Figure 1 caption: “..and then cluster them into …” - > “..and then clusters them into …”\n- Figure 1 caption: “.. the agent use errors of auditory events…” -> “.. the agent uses errors of auditory events…”\n- Figure 2 caption: end the sentence with “respectively”\n- In several places: Forward dynamic network -> forward dynamics network\n- Section 3.2: space missing between “s_v,t” & “and”\n- Section 3.2: “..agent’ actions A_t” -> “..agent’s actions a_t”\n- Section 3.2, paragraph 2, sentence 1: The audio clip is represented by s_t or s_s,t?\n- Section 3.3, paragraph 1, last sentence: “..details of these two-phase below” -> “..details of these two phases below”\n- Section 3.3, Sound clustering: Formally define silhouette score or give an insight for what it captures for the reader.  What is the criteria to determine a new cluster will be created?\n- Section 3.3, Sound clustering: “automatic decide the best K” -> “automatically decide the best K”\n- Section 3.3, Auditory event predictions, last paragraph: “It will reward the agent at the that stage…” This is a grammatically incorrect sentence. Possible correction could be: “It will reward the agent at that stage and encourage it to visit more such states since it is uncertain about this scenario.”\n- Section 3.3, Auditory event predictions, last paragraph: “…avoid dying scenario in the game…” -> “…avoid dying scenarios in the game..”\n- Section 3.3, Auditory event predictions, last paragraph: “.. and keeping seeking novel events…” -> “.. and seeking novel events…”\n- Section 4, paragraph 1: Provide citations for Atari, Habitat and TDW environments.\n- Table 1 caption: “..bold front” -> “..bold font”\n- Section 4.1: Consider formally defining “extrinsic rewards” since it is used several times in the draft.\n- Section 4.1, Atari Game Environment: “..also support an audio API..” -> “..also supports an audio API..” \n- Section 4.1, Atari Game Environment: “..contain the sound effects to compare..” -> “..contain sound effects to compare..” \n- Section 4.1, Atari Game Environment: The last sentence is grammatically incorrect. Consider replacing it with “We follow the standard setting in Pathak et al, 2017; Burda et al., 2018b, where an agent can use external rewards as an evaluation metric to quantify the performance of the exploration strategy. This is because doing well on extrinsic rewards usually requires having explored thoroughly.”\n- Section 4.1, Audio-Visual Explorations on Habitat: “..with Habitat simulator for experiments.” ->  “..with the Habitat simulator for experiments.” \n- Section 4.1, Audio-Visual Explorations on Habitat: “We follows the setting from …” -> “We follow the setting from …”\n- Section 4.1, Audio-Visual Explorations on Habitat: “… can hear the different sound when moving.” -> “…can hear the different sounds when moving.”\n- Section 4.1, Baselines: four baselines are used instead of five as stated in this paragraph.\n- Section 4.1, Baselines: “state-of-the-arts” -> “state-of-the-art”\n- Section 4.1, Implementation details: “..our model use 10K interaction for stage 1…” -> “..our models use 10K interactions for stage 1…” \n- Section 4.2, Result Analysis: What is implied by positive, negative and meaningless sounds? Examples?\n- Section 4.2, Result Analysis: “.. our algorithm works well under what circumstances.” -> “.. under what circumstances our algorithm works well.”\n- Section 4.2, Result Analysis, paragraph 2: “.. event-driven sounds compare with those with action-driven sounds” -> “.. event-driven sounds compared to those with action-driven sounds”\n- Section 4.2, Result Analysis, paragraph 2: “..when the sounds effects mainly consist…” -> “..when the sound effects mainly consist…”\n- Section 4.2, Sound clustering or auditory event prediction and Fig 4.: It is unclear how the Clustering only condition is different from the proposed approach? What is the loss for clustering only that is used as intrinsic reward?\n- Section 4.3, last sentence: Several grammatical errors. Consider replacing with: “We also compute the cluster distances of both models and find that the sound clusters discovered by active exploration are more diverse, thus facilitating the agents to perform in-depth explorations.”\n- Section 4.3, sentence 1: Consider replacing with: “To evaluate the exploration ability of agents trained with our approach, we report the unique state coverage, given a limited exploration budget.\n- Section 4.4, Setup: “The action is repeated 4 times on each frame.” -> “An action is repeated 4 times on each frame”\n- Section 4.4, Result Analysis: “.. intrinsic reward rewards in Figure 7.” -> “..intrinsic rewards in Figure 7.”\n- Section 4.4, Result Analysis: “..prediction errors on latent features space…” -> “..prediction errors on the latent feature space..”\n- Conclusion: “… prediction as an exploration bonuses, which allows RL agent…” -> “… prediction as an exploration bonus, which allows an RL agent…”\n- Conclusion: “Based on the experimental result above, we therefor conclude that sound conveys…” -> “”Based on the experimental results above, we conclude that sound conveys…”\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Investigating using other sensory inputs in our agents, and the impact on exploration is fascinating. We all want to see agents that use more sensory information.

As it stands the paper has several issues that require significant revision, most notably: (1) the polish, quality of the writing and clarity of the text is low, (2) the empirical results are based on 3 runs---at this number we might not have enough data to form valid estimates of the std dev---the error bars are not defined (see Henderson et al 2018), (3) in the ablation studies the hyper-parameters are not tuned (as far as the text suggests) meaning the ablations results might be not representative of the utility of the method, (4) many missing details like hyper-parameter tuning, number of runs in some cases, and reasonable descriptions of experiment protocols and baselines, (5) unsupported claims of causality.

Some of the issues were first raised during the discussion period, so another reviewer was brought in and provided a high quality review with many constructive comments. All reviewers reached clear agreement at the end of the discussion period. 
Decision: Reject