Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: # Summary

This paper investigates the incorporation auditory events into reinforcement learning. Specifically, it proposes a new algorithm that uses event prediction as an intrinsic reward. This algorithm has two phases. In the first phase, an agent is given a small number of episodes to gather diverse auditory data. This phase has two pieces that work in conjunction: 1) As it learns, the agent clusters the sound embeddings it has observed using K-means. 2) To encourage the agent to find diverse auditory data, it is rewarded for reaching states that emit sounds that are far away from its existing cluster centers. In the second  phase (which dominates the first in terms of number of episodes consumed), a new agent is trained to predict the cluster center to which the next auditory event will belong. It is rewarded according to how incorrect its prediction is, thereby encouraging the agent to explore states for which it has difficulty predicting the auditory event.

The paper performs experiments in Atari, Habitat and TDW. It compares against strong vision-based intrinsic reward baselines. It also performs experiments comparing its proposed methodology to ablations with the aim of determining 1) whether it suffices to predict sound features instead of auditory events, 2) whether a two-stage exploration strategy is necessary, 3) whether it is necessary to perform active exploration in the first phase, and 4) whether event classification is necessary.

# Writing

This submission does not read as that of a paper ready for publication. Its organization, unnecessary use of the passive voice, singular-plural inconsistencies, tense mixing, and muddled descriptions weaken its value. Below is a (non-exhaustive!) list of issues.

- Abstract

“We first conduct an in-depth” 

There needs to be a transition from the method description for this “first” to fit here.

- Introduction

Why is Deep Reinforcement Learning all caps?

“algorithms aim to learn a policy of an agent to maximize its cumulative rewards by interacting with environments”

singular/plural

“domains, such as video game”

Game should be plural

“While these results are remarkable, one of the critical constraints is the prerequisite of carefully engineered dense reward signals, which are not always accessible.”

Is Go a good example of this? AlphaZero accomplishes the same task without carefully engineered dense reward signals.

“For example, curiosity-driven intrinsic reward based on prediction error of current (Burda et al., 2018b) or future state (Pathak et al., 2017) on the latent feature spaces have shown promising results.”

This sentence is ordered awkwardly. As written, “have” refers to “curiosity-driven intrinsic reward”, which is singular. “on the latent feature spaces” doesn’t work here.

“visual state is high-dimensional” -> states are

“speech or other nonverbal but audible signals” -> speech or other audible signals OR speech or nonverbal, audible signals

Using “other” here makes it read as if speech is a member of “nonverbal but audible signals”

“However, it is just as much in physics.”

 ?

“A naive strategy would be” -> A naive strategy is

“In the beginning”

Does this mean in the first phase?

“The state that has the wrong prediction is rewarded and encouraged to be visited more.”

Passive voice makes this hard to parse. The agent is the one making predictions and receiving rewards.

“understand our audio-driven exploration works well under what circumstances”

Some words are missing here

“can encourage interest action that involved physical interaction”

Needs fixing

- Related Work

“By leveraging audio-visual correspondences in videos, it can help to learn powerful audio and visual representations through self-supervised learning”

It can help to learn? -> It can learn?

“Reinforcement Learning (RL)”

Acronym has already been introduced.

“makes use of the bootstraps for deep exploration”

bootstraps -> bootstrap

“Here, we mainly focus on the problem of using intrinsic rewards to drive explorations.”

Why is exploration plural?

“The most widely used intrinsic motivation could be roughly divided into two families.” could -> can

The works discussed in the ensuing sentences are already listed and cited in the previous sentences. If you are going to discuss as if you had not already just mentioned them, it is better to exclude them from the previous sentences.
Also, maybe you should add “approaches” or “methods” somewhere in this sentence.

“Burda et al. (2018b) employs the prediction errors of a self-state feature extracted from a fixed and random initialized network and encourage the agent to visit more previous unseen states.” previous -> previously

What is a self-state feature? Burda does not use that term and there is no explanation here.
The end of the sentence doesn’t make sense. What is the subject of “encourage”?

“Another one is the curiosity-based approach (Stadie et al., 2015; Pathak et al., 2017; Haber et al., 2018; Burda et al., 2018a), which is formulated as the uncertainty in predicting the consequences of the agent’s actions.”

-The paper previously said that there were two families so “another one” should be “The other” or something else signifying that this is referring back to the two families comment.

-The paper described family one as a set of approaches (plural) but family two as an approach “singular”.

-The end of the sentence needs fixing.

“The agent is then encouraged to improve its knowledge about the environment dynamics” then -> thereby

Figure 1 would be more clear if it did not use time indexing for the stage 1 section.

“There are numerous works to explore”

that explore?

“More recently, Dhiraj et al. (2020) collected a large sound-action-vision dataset using Tilt-bolt and demonstrates sound signals could provide valuable information for find-grained object recognition”

-Dhiraj et al. is plural so demonstrates should be demonstrate.

-could provide -> can provide

-find-grained -> fine-grained

“More related to us” us -> our work OR this work

“which have shown that the sound signals could provide useful supervisions for imitation learning and reinforcement learning in Atari games.” 

-could -> can

-unnecessary passive voice


Throughout the Sounds and Actions paragraph, the paper repeatedly switches between describing papers in past tense and present tense.

“And then we elaborate on the pipeline of self-supervised exploration through auditory event predictions.” And then -> Then

There is nothing wrong (in general) with starting a sentence with the word “And” but this is not an appropriate place for it.

“a standard Markov Decision Process (MDP), defined as (S, A, r, T , µ, γ). S, A and µ(s) : S → [0, 1] denote”

as -> by

It is bad form to start a sentence with a symbol.

“The transition function T (s 0 |s, a) : S × A × S → [0, 1] defines the transition probability to next-step state s’ if the agent takes action a at current state s.”

The transition function defines this transition probability whether or not the agent executes a at s.

“The goal of training reinforcement learning is to learn an optimal policy π ∗ that can maximize the expected rewards under the discount factor γ as”

This is the goal of reinforcement learning not the goal of training reinforcement learning.

The sentence doesn’t work. It reads “The goal is to learn an optimal policy that is an optimal policy.” Either modify to “The goal is to learn an optimal that is an optimal policy.” or “The goal is to learn an optimal policy. An optimal policy is …” 

“The agent chooses an action a from a policy π(a|s)” from -> according to

“Intrinsic Rewards for Exploration”

 This paragraph is repeating information that was already written

“Designing intrinsic rewards for exploration has been widely used to resolve the sparse reward issues in the deep RL communities”

-Unnecessary passive voice

-What deep RL communities? Isn’t there just one? If there are more than one, what are they?

“transits to the next state with visual observation sv,t+1 and sound effect ss,t+1. “

transits -> transitions

state with -> state, receiving

“We hypothesize that the agents, through this process, could learn the underlying causal structure of the physical world and use that to make predictions about what will happen next, and as well as plan actions to achieve their goals.”

-“and as well as” is redundant

“To better capture the statistic of the raw auditory data”

What statistic?

“For the task of auditory event predictions, perhaps the most straightforward option is to directly regress the sound features Φ(ss,t+1) given the feature embeddings of the image observation sv,t and agent’s actions at”

The paper said this already

“Nevertheless, we find that not very effective. We hypothesize that the main reasons are: 1) the mean squared error (MSE) loss used for regression is satisfied with “blurry” predictions. This might not capture the full distribution over possible sounds and a categorical distribution over clusters; 2) the MSE loss does not accurately reflect how well an agent understands these auditory events. Therefore, we choose instead to define explicit auditory events categories and formulate this auditory event prediction problem as a classification task, similar to (Owens et al., 2016b).”

-Nevertheless is not appropriate here.

-The paper runs this experiment and discusses the results later in the paper. Its confusing to discuss it both as a choice and as an ablation.


“Our AEP framework”

While the acronym can be deduced from the section header, it is still good practice to write it explicitly.

“We need to”

This is a design choice, not a need.

“And then”

And is not appropriate here.

“takes input as the embedding of visual observation and action and predicts which auditory event will happen next.”

needs fixing

“then utilized” -> used

“to explore those auditory events with more uncertainty”

those is unnecessary here

“We will elaborate on the details of these two phases below.”

Get rid of “will”

“The agents start to collect audio data by interacting with the environment”

They start to isn’t good phrasing here. 

“During this exploration, the number of clusters will grow”

fix tense

“After the number of the clusters is saturated”

The description of the sound clustering process is muddled. It does not defined saturated until after it has used it in context.

“To be noted, the number of cluster K is determined automatically in our experiments. In practice, we define K ∈ [5, 30] for clustering at each time step and use the silhouette score to automatically decide the best K, which is a measure of how similar a sound embedding to its own cluster compared to other clusters.”

Do “To be noted” or “In practice” have any added value here?

“we believe it is “saturated””

The paper is defining saturation, it is not a belief.

“We visualize the corresponding visual states in two games (Frostbite and Assault) that belong to the same sound clusters, and it can be observed that each cluster always contains identical or similar auditory events”

-Unnecessary passive voice

-Always is a pretty strong claim. Can we deduce that from this visualization?

“Since we have already explicitly defined the auditory event categories, the prediction problem can then be easily formulated as a classification task”

-Don’t need “already”

-Don’t need “then”

“It will reward the agent at that stage and encourage it to visit more such states since it is uncertain about this scenario”

-Use present tense tense

-at that state?

“In practice, we do find that the agent can learn to avoid dying scenarios in the games since that gives a similar sound effect it has already encountered many times and can predict very well.”

-we do find -> we find

-dying scenarios?

- Experiments

“And then”

And is not appropriate here

“Our primary goal is to investigate whether we could use auditory event prediction as intrinsic rewards to help RL exploration.”

This statement is repeated many times in the paper and has no added value to this section.

“also supports an audio API to provide”

to provide -> that provides

“We use 20 familiar video games”

Is familiar needed here?

“We follow the standard setting in (Pathak et al., 2017; Burda et al., 2018b), where an agent can use external rewards as an evaluation metric to quantify the performance of the exploration strategy. This is because doing well on extrinsic rewards usually requires having explored thoroughly.”

-Passive voice is excessive in multiple places here

-incorrect use of interrogatives

“the the” typo

“Table 1: Categorical results”

These are not results

“could simulate”

can simulate

“physic simulation platform” physics

“Figure 7 ,”comma misplaced

“The agent is required to execute actions to interact with objects of different materials and shapes.”

The agent interacts with objects of different materials and shapes.

“When two objects collide, the environment could generate collision sound based on the physical properties of objects”

could generate collision sound?

“We would like to compare”

We compare

“we choose PPO algorithm”

either choose the PPO algorithm or choose PPO

“The PyTorch implementation”

which?

“the open-source toolbox 1”

-which?

-footnote shouldn’t have space

“As for the auditory prediction network”

Don’t need “as”

“our model use 10K interactions” fix

“previous vision-only intrinsic motivation modules”

don’t need previous here

“We would like to provide an in-depth understanding of under what circumstances our algorithm works well.”

You would like to or you do?

“event-driven sounds which emitted when agents” fix

“action-driven sounds which emitted when agents” fix

“None of the category accounts for the majority” fix

“Since the sound is more observable effects of action” fix

“games dominant with event-driven” fix

“These are also reasonable”

Nothing for “These” to refer to

“Sometimes sound events will occur independently of the agent’s decisions and do not differentiate between different policies”

differentiate is not the right word here

The phrase “ablated study” is used repeatedly. Use “ablation study” instead?

“We further carry out additional experiments”

further unnecessary

“One main contribution of our paper is to use auditory event prediction as an intrinsic reward.”

Repeated without added value

“In Figure 6, using only 16K exploration steps, our agent has already explored all unique states (211 states)” fix

“These results demonstrate that our agent explores environments more quickly and fully, showing the potential ability of exploring the real world.”

Do they? This is a strong claim

“less than 195 unique states” fewer

“These results demonstrate that our agent explores environments more quickly and fully, showing the potential ability of exploring the real world.”

Do they? Again, a strong claim.

There is a sub-header “Setup” of Section 4.4 Experiments on TDW. There is also a TDW sub-header of Section 4.1 Setup.

“The action space consists of moving to eight directions and stop. An action is repeated 4 times on each frame.”fix

“previous vision-based modules”

don’t need previous

“could not” -> did not

“the 3D photo-realistic world, in which a physical event happens.”needs fix

“Instead, our auditory event prediction driven exploration will lead agents”

In constrast, … exploration leads agents

“(See SHE in Figure 8)” see

“We will reward an agent” We reward an agent

“We also want to understand if it is necessary to use audios” audios?

“is powerful for agents to build a causal model of the physical world” fix

- Comments on Organization

-I think it would make more sense to group the question-based analysis together. IE, the ablation studies and the discussions.

-I think whether to use clustering classification or feature prediction should be discussed consistently throughout the paper. As it currently stands, sometimes it is presented as a design choice, sometimes with the claim that the latter is worse (without supporting evidence), and sometimes as a question to be answered by an ablation study.

- Other comments

“As compared to visual cues, sounds are often more directly or easily observable causal effects of actions and interactions.”

Is this obvious?

# Causality?

The paper makes a number of comments about its method learning causal structure. To me, these seem like big claims. The proposed algorithm has no mechanism that tests counterfactuals or, as far as I can tell, any other mechanism for estimating causation, so I see no reason why it would learn anything beyond correlative relationships. Given this fact, in my opinion, if the paper wants to make claims about the fact that its method is learning causal structure, it should back these claims up with experiments.

# Experimental Evaluation

The paper makes very definitive claims (see writing section) about the effectiveness of its method compared to the baselines. In my opinion, there are a number of issues with these claims. First, the paper gives no information (that I could find) about how it tuned the baselines or whether they tuned them at all. Second, for the Atari results, the paper states that it used three random seeds. For the other experiments, it does not say how many seeds it used (or at least I could not find where it said so). Both of these facts make it difficult to know how seriously to take the claims of superior performance.


# Related work

The paper cites many references in its related work section. Yet, I feel that it tells us almost nothing about what it is most important for us to know about:

"More related to us are the papers from Aytar et al. (2018) and Omidshafiei et al. (2018), which have shown that the sound signals could provide useful supervisions for imitation learning and reinforcement learning in Atari games. Concurrent to our work, Dean et al. (2020) uses novel associations of audio and visual signals as intrinsic rewards to guide RL exploration. Different from them, we use auditory event predictions as intrinsic rewards to drive RL explorations."

We are only given a couple of sentences of information about these papers. Additionally, I am not sure that is consistent that the paper claims that Dean is concurrent, but at the same time, designed its experiments to follow Dean “Following (Dean et al., 2020), we use the the apartment 0 in Replica scene (Straub et al., 2019) with the Habitat simulator for experiments.”

# Choice of baselines

The paper appears to be concerned with claiming superior performance over vision-based intrinsic methods, yet it is not clear to me that having superior performance is necessary for the proposed method to have added value, given that the two methods make use of disjointed information for prediction targets. I do not mean to say that these experiments do not have added value—certainly it is nice to see the proposed algorithm compared to existing algorithms—but maybe the adversarial narrative is not the right choice? I understand that the paper is claiming that sounds give the agent better information for intrinsic exploration but, in my opinion, convincing evidence for that claim would require extensive experiments on a wide array of intrinsic methods using both sound and vision and many environments.

# Broader Scope Question

Can a paradigm requiring an initial exploration stage to collect diverse sounds be effective in environments in which some sound events require a large amount learning to discover?

# Closing Thoughts

This seems like promising work, but in my opinion it is not ready for publication. Both the quality of writing and the issues with seeds/tuning independently merit rejection. There are also other issues discussed above.
review_title: Promising but not ready

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces an approach for using auditory event prediction to drive exploration, specifically by using the prediction error of auditory events as a shaping reward which is used along with the extrinsic reward. The approach consists of two phases. In the first phase, the agent explores the environment in a self-supervised manner and collects audio data, which are then clustered. The index of these clusters are the auditory events used during the second phase. In the second phase, an RL agent learns to predict the label of an auditory event at the next time step from the current observation and an action. The prediction error produced from this classification task is given as a shaping reward along with the extrinsic reward to the learning agent. The demonstrated results seem to suggest that this approach can improve over other self-supervised exploration methods in Deep RL.

Pros:
Overall, the approach of using auditory events to drive exploration is novel.
The paper is well-written and presents the idea clearly.

Cons: 
Some implementation details about the approach are not available.
The approach seems to require a pre-training phase that requires collecting a significant amount of data.
Does not provide a clear intuition for why auditory events lead to better performance than other self-supervised approaches.

Questions:
I have a few questions related to the approach, experiment results, and implementation details, which would help clarify my understanding of the paper.
1. How are the intrinsic and extrinsic rewards combined to train an RL agent? The paper doesn’t seem to have any details about this. I assume it is the sum of extrinsic reward and a scaled-down intrinsic reward? If yes, then how was the scaling factor for the intrinsic reward chosen? And how was this chosen for the baseline methods?
2. The approach seems to require a pre-training phase to identify different auditory events through K-means clustering. And this pre-training phase relies on generating behavior to seek out novel clusters. Would it be possible to avoid such a pre-training phase, and instead identify the auditory events as the agent learns to solve the main task? Would be interesting to see if the performance still holds in this scenario.
3. In the results, the amount of data that was used for pre-training doesn’t seem to be included. It seems unfair to the other methods when the introduced method has experienced more data because of the pre-training phase. A fairer comparison would be to train the baseline agents longer to account for the pre-training phase and then make a comparison with the introduced method?
4. In many of the Atari games, the baseline approaches seem to fail in learning? Could the authors comment on why this is the case? 
5. The main experiments (Fig. 3) on Atari seem to use 8 parallel environments. The choice of the number of environments would affect the batch size of the learning update and usually, it is common to use 16 or 32. Why was this choice made?

review_title: Interesting idea. Some implementation details are missing. Intuition is missing.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The manuscript proposes a novel method on how to include audio signals as rewards for image-based reinforcement learning (RL) exploration. More specifically, the agent collects audio signals, identifies the individual audio sources and trains a classifier to forecast the audio cues. The forecasting error in the next time step is used as reward for RL. This encourages the behaviour to explore novel audio cues. Authors’ apply the method to different set of problems (Atari games, Habitat simulator and active learning TDW simulator) and find that the proposed method improves the agent’s behaviour/performance.

Combining audio and video information is essential to improve the explorative behaviour of agents that explore multisensory environments, especially if the relationships between agent behaviour and sounds are causal. In this case, the audio signals provide useful information much earlier in comparison to time than images could. Hence, the proposed approach is relevant for improving the explorative behaviour of agents in multisensory environments. 

The paper is well structured, and the description of the methods is clear. The description of the experiments is not so clear. Note, that the reviewer is not familiar with these environments. For example, in the Habitat experiment, how many sound sources have been used? One in each room? If additional sources were strategically placed in different rooms to favour the proposed method, then one can expect that performance increases. Also, the reviewer found it challenging to interpret the results. For example, Figures 3 shows that the reward of the proposed methods is higher than the reward of other methods. It is not clear, however, how much the performance of the different methods is better than chance level performance. High reward means better performance, however, was the agent able to win game levels? Would be helpful to learn more about how “good” the agent was able to perform the task. Or to clarify which reward level was needed for the agent to successfully explore the environment. Moreover, when assessing the random exploration performance, how was random exploration performed? Each action was randomly selected based on a uniform distribution. If so, then this may not be the most reliable way to estimate random exploration.

Overall, the paper proposes an interesting approach that enable agents to autonomously explore an environment based on unknown auditory cues. 

The pros:
-	Integration of multisensory information in multisensory environments is essential when the information provided by the different sensors is independent and hence contributes to development of more successful behaviour. 
-	The idea to use the prediction error to encourage explorative behaviour is smart
-	Method evaluated in on different environments

The cons: 
-	Interpretation of performance is not straight forward
-	A more systematic analysis of the relationship between relevant sound sources and environment sounds would be helpful to get an idea about the potential and constraints of the proposed method 


### Update after discussion period ###
Good idea, but the results don't clearly support the authors claims. I lowered my score.
review_title: Multisensory approach to explore multisensory environments with some ambiguities

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This work proposes the use of sound prediction as intrinsic reward to guide reinforcement learning (RL) exploration. Sounds are often directly related to causal effects of actions and interactions. By modeling different sound event classes via a clustering algorithm, the cluster prediction errors are used as intrinsic rewards for RL. The proposed approach is tested on three different simulation environments. The results on 22 different environments (20 Atari games, Habitat and TDW environments) illustrate that using exploration capability of the proposed approach and improved performance in comparison to other baseline methods leveraging intrinsic reward modules for RL exploration.

########################

Pros:
- The idea is novel and of interest to the RL community at large. The approach is clear and easy to follow. 
- Comprehensive experimental analysis and convincing results. Specifically, the analysis of dominant sound types in Atari games is well done. It helps understand that the Atari games for which there are no gains with the proposed method have dominantly background sounds. It also helps isolate the games in which sounds are causal effects of actions or events.
- The ablative analysis demonstrates well that sound event prediction is a beneficial way to use audio as intrinsic rewards.
- Testing on different variety of test beds also highlights the exploration capability of the proposed work in comparison to other intrinsic reward methods.

########################

Cons:
- The design choice of using texture features for audio is unclear and not motivated well. What properties do they capture that are desirable for the RL domains used for experimentation? It is also unclear how these are computed. Some more insights about the choice of audio features will be helpful to the reader.
- If distances between sound texture features do not capture causal structure of the auditory events, then why are they suitable to be used for clustering? Wouldn’t the same problem carry over? Why should clusters be assumed to capture inherent causal structure when distances between sound texture features are used to create the clusters in the first place?
- The writing requires more clarity (suggestions below).

########################

Reason for score:
The idea is novel and results presented look great. The writing needs some rework (suggestions for improvements below) along with some explanations for feature computation and experimental conditions. I recommend for acceptance provided the authors revise the manuscript as per requested changes.
---
UPDATE: 
Due to concerns raised by other reviewers, and my own confusion about the computation of audio features, and clarifications on the causality stance, I have lowered my score from 8 to 7. 

########################

Questions during rebuttal:
- Please refer to the questions in the Cons section and other feedback.
- Comparisons are performed with methods using intrinsic motivation modules. How would the proposed work compare with other prior work where audio is used to aid RL (not necessarily as an intrinsic module), such as Aytar et al., 2018; Omidshafiei et al., 2018?


########################

Some typos and other feedback:
- Consider using the \citet command when referring to the authors of a reference paper in a sentence. (Eg: “Silver et al., 2016 concluded that…” versus “(Silver et al., 2016) concluded that …”)
- Introduction, paragraph 1: “…a range of intrinsic reward function.” -> “…a range of intrinsic reward functions.”
- Introduction, last paragraph (prior to bullets): provide citations for the Atari domain, and expand the first usage of TDW.
- Related Work, RL explorations: Tompson sampling -> Thompson sampling
- Related Work, RL explorations: Osband reference missing the year in the references section and the citation.
- Related Work, RL explorations: “…uses the errors of predicting the next state…” -> “use the errors of predicting the next state…”
- Figure 1 caption: “The agent start to collect a diverse set…” -> “The agent starts to collect a diverse set..”
- Figure 1 caption: “..and then cluster them into …” - > “..and then clusters them into …”
- Figure 1 caption: “.. the agent use errors of auditory events…” -> “.. the agent uses errors of auditory events…”
- Figure 2 caption: end the sentence with “respectively”
- In several places: Forward dynamic network -> forward dynamics network
- Section 3.2: space missing between “s_v,t” & “and”
- Section 3.2: “..agent’ actions A_t” -> “..agent’s actions a_t”
- Section 3.2, paragraph 2, sentence 1: The audio clip is represented by s_t or s_s,t?
- Section 3.3, paragraph 1, last sentence: “..details of these two-phase below” -> “..details of these two phases below”
- Section 3.3, Sound clustering: Formally define silhouette score or give an insight for what it captures for the reader.  What is the criteria to determine a new cluster will be created?
- Section 3.3, Sound clustering: “automatic decide the best K” -> “automatically decide the best K”
- Section 3.3, Auditory event predictions, last paragraph: “It will reward the agent at the that stage…” This is a grammatically incorrect sentence. Possible correction could be: “It will reward the agent at that stage and encourage it to visit more such states since it is uncertain about this scenario.”
- Section 3.3, Auditory event predictions, last paragraph: “…avoid dying scenario in the game…” -> “…avoid dying scenarios in the game..”
- Section 3.3, Auditory event predictions, last paragraph: “.. and keeping seeking novel events…” -> “.. and seeking novel events…”
- Section 4, paragraph 1: Provide citations for Atari, Habitat and TDW environments.
- Table 1 caption: “..bold front” -> “..bold font”
- Section 4.1: Consider formally defining “extrinsic rewards” since it is used several times in the draft.
- Section 4.1, Atari Game Environment: “..also support an audio API..” -> “..also supports an audio API..” 
- Section 4.1, Atari Game Environment: “..contain the sound effects to compare..” -> “..contain sound effects to compare..” 
- Section 4.1, Atari Game Environment: The last sentence is grammatically incorrect. Consider replacing it with “We follow the standard setting in Pathak et al, 2017; Burda et al., 2018b, where an agent can use external rewards as an evaluation metric to quantify the performance of the exploration strategy. This is because doing well on extrinsic rewards usually requires having explored thoroughly.”
- Section 4.1, Audio-Visual Explorations on Habitat: “..with Habitat simulator for experiments.” ->  “..with the Habitat simulator for experiments.” 
- Section 4.1, Audio-Visual Explorations on Habitat: “We follows the setting from …” -> “We follow the setting from …”
- Section 4.1, Audio-Visual Explorations on Habitat: “… can hear the different sound when moving.” -> “…can hear the different sounds when moving.”
- Section 4.1, Baselines: four baselines are used instead of five as stated in this paragraph.
- Section 4.1, Baselines: “state-of-the-arts” -> “state-of-the-art”
- Section 4.1, Implementation details: “..our model use 10K interaction for stage 1…” -> “..our models use 10K interactions for stage 1…” 
- Section 4.2, Result Analysis: What is implied by positive, negative and meaningless sounds? Examples?
- Section 4.2, Result Analysis: “.. our algorithm works well under what circumstances.” -> “.. under what circumstances our algorithm works well.”
- Section 4.2, Result Analysis, paragraph 2: “.. event-driven sounds compare with those with action-driven sounds” -> “.. event-driven sounds compared to those with action-driven sounds”
- Section 4.2, Result Analysis, paragraph 2: “..when the sounds effects mainly consist…” -> “..when the sound effects mainly consist…”
- Section 4.2, Sound clustering or auditory event prediction and Fig 4.: It is unclear how the Clustering only condition is different from the proposed approach? What is the loss for clustering only that is used as intrinsic reward?
- Section 4.3, last sentence: Several grammatical errors. Consider replacing with: “We also compute the cluster distances of both models and find that the sound clusters discovered by active exploration are more diverse, thus facilitating the agents to perform in-depth explorations.”
- Section 4.3, sentence 1: Consider replacing with: “To evaluate the exploration ability of agents trained with our approach, we report the unique state coverage, given a limited exploration budget.
- Section 4.4, Setup: “The action is repeated 4 times on each frame.” -> “An action is repeated 4 times on each frame”
- Section 4.4, Result Analysis: “.. intrinsic reward rewards in Figure 7.” -> “..intrinsic rewards in Figure 7.”
- Section 4.4, Result Analysis: “..prediction errors on latent features space…” -> “..prediction errors on the latent feature space..”
- Conclusion: “… prediction as an exploration bonuses, which allows RL agent…” -> “… prediction as an exploration bonus, which allows an RL agent…”
- Conclusion: “Based on the experimental result above, we therefor conclude that sound conveys…” -> “”Based on the experimental results above, we conclude that sound conveys…”

review_title: Official Blind Review #1

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Update**: Other reviewers have pointed out issues with this paper's ablation study. Additionally, it is difficult to trust the empirical results because they are based on only three runs. In light of these criticisms, I have updated my score from a 7 to a 5. I still think this idea is neat and am generally a proponent of introducing audio into work on RL, but the experiments as presented in this submission do not currently paint a complete picture.

**Summary**: This paper investigates sound perception as a means of intrinsic reward for reinforcement learning agents. Specifically, the proposed method rewards the agent for discovering state/action pairs which lead to sound events that are difficult to predict. Especially in environments where sound effects are correlated with high-level events, this strategy tends to improve performance over sound-agnostic baselines.

Overall, I think this is an interesting paper. Audio events provide important queues which help humans understand the natural world and influence their decision. Such rich acoustic structure is often imitated in simulated environments, even in early Atari games. While substantial effort has gone into extracting as much usable information as possible out of the pixels of Atari games, the easily-accessible sound systems in these environments have been mostly ignored. It stands to reason that, especially for particular games, this information could be helpful.

Hypothesizing that encountering novel audio events may be informative for learning good policies, the authors propose a strategy which explicitly rewards agents for finding such states. They show that this strategy leads to improved performance on several environments, especially ones where audio information is associated with high-level events and interactions (e.g. balls colliding) as opposed to background noise (e.g. music in games).

From an audio perspective (my primary area of expertise), the proposed strategy seems reasonable. The authors cluster perceptually-informed embeddings of audio slices to identify discrete classes, which makes sense when targeting environments whose audio primarily consists of sound effects (e.g. a clacking sound) synchronized to high-level events (e.g. two balls colliding). Why did the authors choose to use a texture-based sound embedding (McDermott & Simoncelli, 2011) as opposed to something more standard like log-amplitude Mel spectrograms? I would be quite interested to see how the performance of the latter compares to texture-based embeddings.

The experiments also seem reasonable: they compare the performance of the same policy-learning model/algorithm with different intrinsic reward sources on a slew of sparse-reward environments. The results seem to indicate that the proposed intrinsic reward usually leads to the best policy among the examined sources of intrinsic reward. One criticism is that it would be nice to see if these different intrinsic reward strategies are symbiotic; i.e., if multiple sources can be combined to improve performance. It would also have provided stronger evidence in favor of the proposed intrinsic reward function to see results on different policy-learning algorithms (besides PPO and CNNs). A caveat here is that I have limited experience on RL and defer to the expertise of the other reviewers in assessing the relevance of the baselines (ICM, RND, RFN, DIS).

The authors need to do more to distinguish their work from that of Omidshafiei et al. 2018. In a cursory overview of that work, it appears that there is a lot of overlap with the proposed work. It seems that the primary distinction is that the proposed work uses audio _only as part of the intrinsic reward_ (leaving the policy model unaware of audio cues), while the prior work _adds audio processing to the agent_. Can the authors comment further on the distinctions between their work and this prior work? Also, it's not immediately clear from the paper if the policy model (agent) receives audio as part of its state input; can the authors clarify?
review_title: Removing the earplugs from RL

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper proposes a novel type of intrinsic rewards for RL agents based on audio events prediction. A two stage method is proposed that includes sound clustering and auditory event prediction. The auditory event prediction error is used as an intrinsic reward for better exploration of the agents. The idea is evaluated on Atari games, and two 3D simulators: Habitat and TDW.

Strengths:
1. Decent idea to use audio as curiosity to drive the exploration of an agent.  
2. Well motivated by how humans integrate multisensory inputs to understand the physical world.

Weakness:
1. Justification of causal effect of its actions. See detailed comments.
2. Why not using bottom up clustering methods such as Agglomerative Clustering so that the number of clusters is not needed to be specified? The current method mannually set K to be in the range of 5 - 30, and then decide the best K is somewhat cubersome.
3. Most of the results are using Atari games as proof of concept. The results on more realistic environements are not very convincing. What are the sound events in HABITAT? If an agent just navigates in the environment listening to the same sound, how it's going to help exporation at all? There is no event defined.
4. The task is set up as an event classification task to get the intrinsic reward, so audio is actually not directly used. Technically, audio might not even be necessary in this case. What if directly using the meta data from the graphical engine to know the sound type or event type and use the ground-truth to guide the agent's learning process? This can serve as an upper bound for the proposed method to cluster audio events. Or, what if clusering the visual frames to get the event type? These are all useful baselines to demonstrate the use of audio is essential.

Detailed Comments/Questions:
1. The paper repetitively mention the agent is encouraged to understand the causal effect of its actions. However, the sound in experiments are usually just accompanying a visual event. Is it just audio-visual correspondence that helps or causality?
2. Comparison to Dean et al. 2020 should be more clear. It is mentioned this work mainly studied if the sound signals along could be utilized as intrinsic rewards, which is confusing. What is the difference compared to Dean et al. 2020 is still not clear.
3. In figure 4, how to directly use sound clustering as an intrinsic reward is not clear to this reviewer.
4. An informative baseline would be to remove clusters, but set up the task as a binary classification task: making sound or not making sound. It would be informative to know whether the agent is really leveraging the knowledge of different sound events for exploration.
5. Plenty of typos and grammar mistakes should be fixed.
    - P2, RL Explorations, A separate line of work studies adopt?
    - Sec. 3.2, to represent each audio clip s_t s_s,t ...?
    - Sec. 4.1, how well an exploration stragegy is ;  We follows -> We follow; State-of-the-arts
    ........

Justification of recommendation:
This paper proposes a nice idea, but the paper is not very well written. The experiments needs further clarifications. This reviewer is happy to raise the score if these concerns can be addressed in authors' response.




###Final Recommendation###

Based on the discussions with other reviewers and AC, this paper is not ready to publish at this stage mainy due to the following reasons:
1. the big claim of causality as also pointed out by R6
2. the writing should be significantly improved and the experiments lack details as pointed out by all reviewers
3. the new problems found during discussion with the AC regarding the ablation study, and seeds, etc. 

In summary, this paper presents an interesting idea, but the experiments and writing in its current shape make the paper insufficient to be published at ICLR. The authors are encouraged to polish the paper in writing and experiments for future resubmissions.
review_title: The paper has good idea with somewhat unconvincing experiments.

Meta-review: Investigating using other sensory inputs in our agents, and the impact on exploration is fascinating. We all want to see agents that use more sensory information.

As it stands the paper has several issues that require significant revision, most notably: (1) the polish, quality of the writing and clarity of the text is low, (2) the empirical results are based on 3 runs---at this number we might not have enough data to form valid estimates of the std dev---the error bars are not defined (see Henderson et al 2018), (3) in the ablation studies the hyper-parameters are not tuned (as far as the text suggests) meaning the ablations results might be not representative of the utility of the method, (4) many missing details like hyper-parameter tuning, number of runs in some cases, and reasonable descriptions of experiment protocols and baselines, (5) unsupported claims of causality.

Some of the issues were first raised during the discussion period, so another reviewer was brought in and provided a high quality review with many constructive comments. All reviewers reached clear agreement at the end of the discussion period. 
Decision: Reject