SOUND ADVERSARIAL AUDIO-VISUAL NAVIGATION
Year: 2022

Abstract:
Audio-visual navigation task requires an agent to find a sound source in a realistic, unmapped 3D environment by utilizing egocentric audio-visual observations. Existing audio-visual navigation works assume a clean environment that solely contains the target sound, which, however, would not be suitable in most realworld applications due to the unexpected sound noise or intentional interference. In this work, we design an acoustically complex environment in which, besides the target sound, there exists a sound attacker playing a zero-sum game with the agent. More specifically, the attacker can move and change the volume and category of the sound to make the agent suffer from finding the sounding object while the agent tries to dodge the attack and navigate to the goal under the intervention. Under certain constraints to the attacker, we can improve the robustness of the agent towards unexpected sound attacks in audio-visual navigation. For better convergence, we develop a joint training mechanism by employing the property of a centralized critic with decentralized actors. Experiments on two real-world 3D scan datasets, Replica, and Matterport3D, verify the effectiveness and the robustness of the agent trained under our designed environment when transferred to the clean environment or the one containing sound attackers with random policy. Project: https://yyf17.github.io/SAAVN.

heading: 1 INTRODUCTION
text: Audio-visual navigation, as currently a vital task for embodied vision (Gordon et al., 2018; Lohmann et al., 2020; Nagarajan & Grauman, 2020), requires the agent to find a sound target in a realistic and unmapped 3D environment by exploring with egocentric audio-visual observations (Chen et al., 2019; Gupta et al., 2017; Chaplot et al., 2020). Inspired by the simultaneous usage of eyes and ears in human exploration (Wilcox et al., 2007; Flom & Bahrick, 2007), audio-visual association benefits the learning of the agent (Dean et al., 2020). A recent work Look, Listen, and Act (LLA) has proposed a three-step navigation solution of perception, inference, and decision-making (Gan et al., 2020). SoundSpaces is the first work to establish an audio-visual embodied navigation simulation platform equipped with the proposed Audio-Visual embodied Navigation (AVN) baseline that resorts to reinforcement learning (Chen et al., 2020). In response to the long-term exploration problem that is caused by the large layout of the 3D scene and the long distance to the target place, Audio-Visual Waypoint Navigation (AV-WaN) proposes an audio-visual navigation algorithm by setting waypoints as sub-goals to facilitate sound source discovering (Chen et al., 2021b). Besides, Semantic Audio-Visual navigation (SAVi) develops a navigation algorithm in a scene where the target sound is not periodic and has a variable length; that is, it may stop during the navigation process (Chen et al., 2021a).
Despite the fruitful progress in audio-visual navigation, existing works assume an acoustically simple or clean environment, meaning there is no other sound but the source itself. Nevertheless, this
∗Corresponding author: Fuchun Sun.
assumption is hardly the case in real life. For example, a kettle in the kitchen beeps to tell the robot that the water is boiling, and the robot in the living room needs to navigate to the kitchen and turn off the stove; while in the living room, two children are playing a game, chuckling loudly from time to time. Such an example poses a crucial challenge on current techniques: can an agent still find its way to the destination without being distracted by all non-target sounds around the agent? Intuitively, the answer is no if the agent has not been trained in acoustically complex environments as in the example listed above. Although the answer is no, this ability is what we expect the agent to possess in real life.
In light of these limitations, we propose first to construct such an acoustically complex environment. In this environment, we add a sound attacker to intervene. This sound attacker can move and change the volume and type of the sound at each time step. In particular, the objective of the sound attacker is to make the agent frustrated by creating a distraction. In contrast, the agent decides how to move at every time step, tries to dodge the sound attack, and explores for the sound target well under the sound attack, as illustrated in Fig. 1. The competition between the attacker and the agent can be modeled as a zero-sum two-player game. Notably, this is not a fair game and is more biased towards the agent for two reasons. First, the sound attack is just single-modal and will not intervene in any visual information obtained by the agent. Second, as will be specified in our methodology, the sound volume of the attacker is bounded via a relative ratio (less than 1) of the sound target. We anticipate constraining the attacker’s power and encouraging it to intervene but not defeat the agent with these two considerations. With such a design, we can improve the agent’s robustness between the agent and the sound attacker during the game. On the other hand, our environment is more demanding than reality since there are few attackers in our lives. Instead, most behaviors, such as someone walking and chatting past the robot, are not deliberately embarrassing the robot but just a distraction to the robot, exhibiting weaker intervention strength than our adversarial setting. Even so, our experiments reveal that an agent trained in a worst-case setting can perform promisingly when the environment is acoustically clean or contains a natural sound intervenor using a random policy. On the contrary, the agent trained in a clean environment becomes disabled in an acoustically complex environment.
Our training algorithm is built upon the architecture by (Chen et al., 2020), with a novel decisionmaking branch for the attacker. Training two agents separately (Tampuu et al., 2017) leads to divergence. Hence we propose a joint Actor-Critic (AC) training framework. We define the policies for the attacker based on three types of information: position, sound volume, and sound category. Exciting discoveries from experiments demonstrate that the joint training converges promisingly in contrast to the independent training counterpart.
This work is the first audio-visual navigation method with a sound attacker to the best of our knowledge. To sum up, our contributions are as follows.
• We construct a sound attacker to intervene environment for audio-visual navigation that aims to improve the agent’s robustness. In contrast to the environment used by prior experiments (Chen et al., 2020), our setting better simulates the practical case in which there exist other moving intervenor sounds.
• We develop a joint training paradigm for the agent and the attacker. Moreover, we have justified the effectiveness of this paradigm, both theoretically and empirically.
• Experiments on two real-world 3D scenes, Replica (Straub et al., 2019) and Matterport3D (Chang et al., 2017) validate the effectiveness and robustness of the agent trained under our designed environment when transferred to various cases, including the clean environment and the one that contains sound attacker with random policy.

heading: 2 RELATED WORK
text: We introduce the research related to our work, including audio-visual navigation, adversarial training in RL, and multi-agent learning.
Audio-visual embodied navigation. This task demands a robot equipped with a camera and a microphone to interact with the environment and navigate to the source sound. Existing algorithms towards this task can be divided into two categories according to whether topological maps are constructed or not. For the first category (Gan et al., 2020; Chen et al., 2021b), LLA (Gan et al., 2020)
plans the robot’s navigation strategy based on graph-based shortest path planning. Following LLA, AV-WaN (Chen et al., 2021b) combines dynamically setting waypoints with the shortest path algorithm to solve the long-period audio-visual navigation problem. The second category of methods works in the absence of a topological map (Chen et al., 2021a; 2020). In particular, SAVi (Chen et al., 2021a) aims to solve the audio-visual navigation problem with temporary sound source by introducing the category information of the source sound and pre-training the displacement from the robot to the sound source; AVN (Chen et al., 2020) constructs the first audio-visual embodied navigation simulation platform—SoundSpaces and makes use of Reinforcement Learning (RL) for the training of the navigation policy. As presented in the Introduction, all environments used previously (including SoundSpaces) assume clean sound sources, which is hardly the case in real and noisy life. By contrast, we build an acoustically complex environment that allows a sound attacker to move and change the volume and category of sound at each time step in an episode. In this environment, we train the navigation policy of the agent under the sound attack, which delivers a more robust solution in real applications.
Adversarial attacks and adversarial training in RL. In general, adversarial attacks (Tian & Xu, 2021) /training in RL are divided into two classes: learning to attack (Huang et al., 2017; Kos & Song, 2017; Lin et al., 2017; Pattanaik et al., 2018) and learning to defence (Pinto et al., 2017; Gleave et al., 2020; Zhang et al., 2020) that targets on learn a robustness policy by state adversarial through external forces or sensor noise. Our paper falls into the second class. A close work to our method is State Adversarial MDP (SA-MDP) (Zhang et al., 2020) that leverages sensor noise in vision to improve the robustness of the algorithm. The main difference between our method and SA-MDP is that our method precisely permits a sound attacker to move in the room and employs the resulting sound as a distractor, while in SA-MDP, the adversarial state is created arbitrarily and thus not necessarily fits the actual scene. Besides, SA-MDP initially copes with the visual modality; hence, we have changed SA-MDP for the attack of the sound modality for the comparison with our method (See § 4).
Multi-agent learning. We design two frameworks similar to independent learning (Tampuu et al., 2017) and multi-agent learning (Sunehag et al., 2018). However, the framework of independent learning does not converge (See § 4). We formulate our learning algorithm as a two-player game employed the property of a centralized critic with decentralized actors (Wang et al., 2021) to guarantee the training convergence of our method in theory (See Theorem 1).

heading: 3 SOUND ADVERSARIAL AUDIO-VISUAL NAVIGATION

heading: 3.1 PROBLEM DEFINITION
text: Problem modeling of ours. We model the agent as playing against an attacker in a two-player Markov game (Simon, 2016). We denote the agent and attacker by superscript ω and ν , respectively. The gameM = (S, (Aω,Aν),P, (Rω,Rν)) consists of state set S, action sets Aω and Aν , and a joint state transition functionP : S×Aω×Aν → S . The reward functionRω : S×Aω×Aν×S → R for agent and Rν : S × Aω × Aν × S → R for attacker respectively depends on the current state, next state and both the agent’s and the attacker’s actions. Each player wishes to maximize their discounted sum of rewards, where Rω = r, Rν = −r. r is the reward given by the environment at every time step in an episode. Our problem is modeled as following (See Fig. 2(c)):
π⋆ = (π⋆,ω, π⋆,ν) = argmax πω∈Πω {argmin πν∈Πν {G(πω, πν , r)}} (1)
Inspired by value decomposition networks (Sunehag et al., 2018) and QMIX (Rashid et al., 2018), we design G(πω, πν , r) as Equation (2), where G(πω, r) and G(πν , r) are expected discounted, cumulative rewards of the agent and the attacker respectively.
G(πω, πν , r) = [G(πω, r), G(πν , r)] (2)
SA-MDP. As a reference, we introduce the previous adversarial MDP proposed by (Zhang et al., 2021) as well. In SA-MDP (See Fig. 2 (b)), we optimize δadv := argmax
δ:∥δ∥≤ϵ DKL[π
ω(a|s) | πω(a|s+
δ)]. Intuitively, the state of SA-MDP is on the ball with radius ϵ.

heading: 3.2 THEORY ANALYSIS
text: Why does our model and training mechanism work? Through theoretical analysis, we were pleasantly surprised to find that the observation space of a sound attacker is bounded in the projection space. For more details, see Theorem 1.
Notations. F , ψ stands for short-time Fourier transform and the room impulse response, respectively. ςg and ςν stand for the waveform signal received by the robot came from a sound source and sound attacker at every time step, respectively. Ig and Iν stand for the egocentric visual received by the robot in a clean environment and an acoustically complex environment, respectively. O and O′ stand for the observation of what the robot received in a clean environment and an acoustically complex environment, respectively. O and O′ is defined as follows: O = [Ig,F(ψg ∗ ςg)] , O′ = [Iν ,F(ψg∗ςg+α·ψν ∗ςν)] , where · stands for multiply, ∗ stands for time domain convolution, α is the value of action aν,vol , ϵ is a parameter.
Sound attackers do not intervene in the robot’s visual modality observation in an acoustically complex environment. So we have :
Property 1 Ig = Iν at every time step in an given episode.
Assumption 1 The energy of the sound source sent to a robot at every time step for a fixed duration from the same sound set has an upper bound e , ∥F(ςν)∥22 ≤ e , ∥F(ςg)∥22 ≤ e.
The distance of intervention is a map function of O and O′ : ∆ : O × O′ → R. We give a distance definition of O and O′ as following:
Definition 1 The distance is : ∆(O,O′) = ∥Ig − Iν∥22 + ∥F(ψg ∗ ςg + α ·ψν ∗ ςν)−F(ψg ∗ ςg)∥22
Then the sound attacker’s observation space Bϵ(O) in an acoustically complex environment based on the above intervention distance is formalized as follows: Bϵ(O) = {O′ : ∆(O,O′) ≤ ϵ}.
Theorem 1 The observation space of the sound attacker is bounded in the projection space. If a sound attacker’s observation is projected into a frequency domain space, the projected result is in a sphere with a radius of ϵ.
Proof 1 ∆(O,O′) = ∥Ig − Iν∥22 + ∥F(ψg ∗ ςg + α · ψν ∗ ςν)−F(ψg ∗ ςg)∥22 (Def. 1)
= ∥F(ψg ∗ ςg) + F(α · ψν ∗ ςν)−F(ψg ∗ ςg)∥22 (Prop. 1, P rop. 2) = α · ∥F(ψν ∗ ςν)∥22 = α · ∥F(ψν) · F(ςν))∥22 (Prop. 2) = α/2π · ∥F(ςν)∥22 (Prop. 2) ≤ α·e/2π = ϵ (Assump. 1) (3)
Theorem 1 shows that the attack is bounded and guarantees the rationality of our algorithm. Property 2 in appx. B provides the linear property of the Fourier transform, the convolution theory of the Fourier transform, and the Fourier transform of the Dirac delta function.

heading: 3.3 SOUND ADVERSARIAL AUDIO-VISUAL NAVIGATION
text: We propose Sound Adversarial Audio-Visual Navigation (SAAVN), a novel model for the audiovisual embodied navigation task. Our approach is composed of three main modules (Fig. 3). Given visual and audio inputs, our model 1) encodes these cues and make a decision for the motion of the agent, then 2) encodes these cues and decide how to act for the sound attacker to make an acoustically complex environment, and finally 3) make a judgment for the agent and the attacker and to optimization. The agent and the attacker repeat this process until the agent has been reached and executes the Stop action.
Environment. Our work is based on the SoundSpaces (Chen et al., 2020) platform and Habitat simulator (Savva et al., 2019) and with the publicly available datasets: Replica (Straub et al., 2019) and Matterport3D (Chang et al., 2017) and SoundSpaces audio dataset. In SoundSpaces, the sound is created by convolving the selected audio with the corresponding binaural room impulse responses (RIRs) under one of the directions. When a sound attacker emits a chosen sound from its position, the emitted omnidirectional audio is convolved with the corresponding binaural RIR to generate a binaural response from the environment heard by the agent when facing each direction. In this sense, the attacker’s sound also considers the reflections on the surface of objects in the environment, making it physically admissible and realistic. The agent’s reward is based on how close the robot is away from the goal and whether it succeeds in reaching it. The setting is the same as of the SoundSpaces. The action space of the agent is navigation motions, which is the same as the setting of the SoundSpaces. An environment attacker embodied in the environment must take actions from a hybrid action space Aν . For brevity, the abbreviation of superscripts position, volume, and category are set to pos, vol, and cat, respectively. The hybrid action space is the Cartesian product of navigation motions space Aν,pos, volume of sound space Aν,vol and category of sound space Aν,cat: Aν = Aν,pos ×Aν,vol ×Aν,cat . For more details, see appx. C. Perception, act, and optimization. Our model uses acoustic and visual cues in the 3D environment for efficient navigation. Our model has mainly comprised of three parts: the environment attacker, the agent, and the optimizer (See Fig. 3). At every time step t, the agent and the attacker receives an observation Ot = (It, Bt), where I is the egocentric visual observation consisting of an RGB and a depth image; B is the received binaural audio waveform represented as a two-channel spectrogram. Our model encodes each visual and audio observation with a CNN, respectively, where the output of each CNN are visual vector fI1(It) and audio vector fB1(Bt). Then, we concatenate the two vectors to obtain observation embedding representation e1 = [fI1(It), fB1(Bt)]. We transform observation embedding representation to calculate state representation by a gated recurrent unit (GRU), s1t = GRU(e 1 t , h 1 t−1). An actor-critic network uses s 1 t to predict the action distribution πωθ (a ω t |s1t , h1t−1) and value of the state V ωθ (s1t , h1t−1). We also encode visual and audio observation with a CNN for environment attacker, where the output of each CNN are vectors fI2(It) , fB2(Bt). We then concatenate the two vectors to obtain observation embedding representation e2 = [fI2(It), fB2(Bt)]. We also transform observation embedding representation to calculate state representation by a GRU, s2t = GRU(e 2 t , h 2 t−1) . Three actor-critic networks use s 2 t to predict the action distribution: πν, posθ (a ν, pos t |s2t , h2t−1) , π ν, vol θ (a ν, vol t |s2t , h2t−1) , π ν, cat θ (a ν, cat t |s2t , h2t−1) and value of the state: V ν, posθ (s 2 t , h 2 t−1) , V ν, vol θ (s 2 t , h 2 t−1) , V ν, cat θ (s 2 t , h 2 t−1) . All actors and critics are modeled by a single linear layer neural network, respectively. Finally, four action samplers sample the next action aωt , a ν, pos t , a ν, vol t , a ν, cat t from these action distributions generated by AgentActor,
PositionActor, VolumeActor and CategoryActor respectively, determining the agent’s next motion in the 3D scene. The total critic is a linear sum of PositionCritic, VolumeCritic, and CategoryCritic (For weights details, see appx. G ). The agent and the environment attacker optimize their expected discounted, cumulative rewards G(πω, r) and G(πν , r) respectively. The loss of each branch actorcritic network and the total loss of our model as Equation (4).
Lj = ∑ 0.5 · (V̂θj (s)− V j(s))2 − ∑ [Âj log(πθj (a | s)) + β ·H(πθj (a | s))]
Lν = 1/3 · (Lν,cat + Lν,vol + Lν,pos) L = 1/6 · Lν,cat + 1/6 · Lν,vol + 1/6 · Lν,pos + 1/2 · Lω
(4)
where j ∈ {(ν, cat), (ν, vol), (ν, pos), (ω)} . V̂θj (s) is estimated state value of the target network for j . Vj(s) = max
a∈Aj E[rt + γ · Vj(st+1) | st = s] . Âjt =
∑T−1 i=t γ i+2−t · δji is the advantage for a
given length-T trajectory and δjt = rt + γ · Vj(st+1) − Vj(st) . We optimize the objective follows from Proximal Policy Optimization (PPO) (Schulman et al., 2017) .
Algorithms. Our algorithms are as the following:
Algorithm 1: Sound Adversarial Audio-Visual Navigation Data: Environment E , stochastic policies ω and ν, initial parameters θω0 for ω and θν0 for ν, number of updates Niter, N . Result: θωNiter , θ ν Niter
1 for i=1, 2, ... Niter do 2 // Run policy πθωi−1 and πθνi−1 in environment for N episodes T time steps ; 3 {(ot,i, ht−1,i, aωt,i, aνt,i, rωt,i, rνt,i)} ← roll(E , πθωi−1 , πθνi−1 , T ) ; 4 Compute advantage estimates Âω1 , · · · , ÂωT , Âν1 , · · · , ÂνT ; 5 // Optimize L(Equation 4) w.r.t. θω and θν ; 6 θωi , θ ν i ← policy optimize with PPO ; 7 end

heading: 4 EXPERIMENTS
text: Task. We use AudioGoal navigation (Chen et al., 2020) tasks to test the acoustically complex environment we designed. In this task, the robot moves in a 3D environment. At each time step t, it obtains an RGB and a depth image from its camera, and the binaural microphone receives sensor observations to get Ot. As the robot starts navigating, it does not know the scene map; it
must accumulate observations to understand the geometry of the scene during navigation. The robot should use the sound from the audio source to locate successfully and navigate to the target.
Baselines. We compare our model to the following baselines and existing works:
1. Random: A random policy that uniformly samples one of three actions and executes Stop automatically when it reaches the goal (perfect stopping).
2. AVN (Chen et al., 2020): An audiovisual embodied navigation trained in an environment without sound intervention.
3. SA-MDP (Zhang et al., 2020): A model aims to improve the robustness by state adversarial. We adopt its idea but only intervene state of the sound input and do not process the visual information.
Metrics and symbols. The evaluations are compared by the following navigation metrics: 1) success weighted by inverse path length (SPL): the standard metric (Anderson et al., 2018) that weighs successes by their adherence to the shortest path; 2) Agent’s average episode reward: Rmean. The detailed definitions of the symbols used later in this section are as follows: (1) πν,pos, πν,vol and πν,cat are abbreviated as P, V and C respectively. (2) X , Xrandom and Xfix denote policies, where X ∈ {P, V,C} is a policy to learn, Xrandom is a random policy, and when X ∈ {P}, Xfix is a determined policy which acts a constant action after random initialization in an episode, while X ∈ {V,C}, Xfix is a determined policy which act a constant action set in advance in all episodes. (3) X,Yfix is abbreviated as X , where X is a policy, and Y ∈ {{P, V,C} \ {X}} are set to a determined policy Yfix. (4) SAAVN:X stands for a model variant, where X is a policy for πν and X ∈ {P, V,C}. (5) SAAVN:X,Y=0.1 is a model variant named SAAVN:X, and Yfix is set to 0.1, where X ∈ {P, V,C}, Y ∈ {{P, V,C} \ {X}}. (6) SAAVN:X,Y=0.1,Z=person6 is a model variant named SAAVN:X, where Yfix is set to 0.1, Zfix is set to person6, X ∈ {P, V,C}, Y ∈ {{P, V,C} \ {X}}, Z ∈ {{P, V,C} \ {X,Y }}. See appx. H and F for more details.
Navigation results. We have built a large number of complex environments with sound attackers therein. The baselines and our SAAVN have been tested several times with different seeds in all these acoustically complex and clean environments to obtain the average navigation ability under each environment. It can be seen from Table 1, 2 that our model achieves the best navigation capabilities in all environments. For more experiments results, see appx. H.
Trajectory comparisons. Fig. 4 shows the test episodes for our SAAVN model. The environment of each row in the figure is consistent, and the model of each column is too. SA-MDP failed to complete the task successfully in all two environments. AVN can complete the task in a clean environment but fails in an acoustically complex environment. SAAVN completed the tasks in all two environments. What is more, the navigation track of SAAVN in an environment without intervention is shorter than that of AVN. These fully demonstrate the navigation capabilities of SAAVN. For more trajectory comparisons, see appx. I.
Robustness of the model. In order to verify the robustness of our algorithm, we designed 5 sound attackers, with the settings as follows: πν,pos is a learned policy, πν,cat is set to person6, while πν,vol is fixed to 0.1, 0.3, 0.5, 0.7 and 0.9 respectively. We train AVN, SA-MDP, and SAAVN named SAAVN: P, V=0.1, C=person6 respectively in the same environment created by a designated sound attacker. Then we test them multiple times in the above five environments. Fig. 5 shows the performance of different models under different sound attacks. We found that these sound attackers have certain attack capabilities against these three algorithms. However, it is found that our method’s performance decreases more slowly, which fully demonstrates that our method helps to improve the robust performance of the algorithm.
Is the louder the sound attacker’s volume, the better? To verify the navigation ability of an agent that has grown in an acoustically complex environment in a clean environment, we first train our model in the acoustically complex environments where the sound attacker exists and then test the navigation ability of the model by migrating to the environment which the sound attacker is removed. In Fig. 6, SAAVN : P, V=0.1 is short for a model variant named SAAVN : P, V=0.1, C=person6. Others are similar. It can be seen from Fig. 6 that the navigation ability of the attacker is excellent when the volume of the attacker is 0.1 and 0.9, but not very good when the volume is 0.5. The ablation experiments show that simply increasing the sound volume of the attacker does not necessarily lead to better performance. The relationship between the navigation capacity and the volume of the sound attacker is not straightforward and depends on other factors, including the position and sound category. It hence supports the necessity of our method to make the volume policy of the attacker learnable. As such, agents can learn better navigation skills in an acoustically intervened environment.
Navigation results in the clean environment. When the sound attacker does not exist, what is the navigation ability of an agent that grows in an acoustically complex environment? The performance of AVN is trained in a simple environment and tested in a clean environment; The performances of SAAVN are trained in acoustically complex environments and tested in a clean environment. It can be seen from Fig. 7 (a) that the ability of an agent that grows up in an acoustically complex
environment to navigate in a clean environment depends on the complexity of the environment. If the environment is acoustically complex, but within a certain range, the ability of the agent will increase; if the environment is too complicated and the changes are relatively great, the navigation ability of the agent is reduced a bit, but not too much. The ablation study shows that the environment should not be too complicated to achieve optimal navigation capabilities.
Navigation results in acoustically complex environments. What is the navigation ability of an agent in an acoustically complex environment? In Fig. 7 (b), we can see different sound attackers. As the attack intensity of sound attackers ascends, the navigation capabilities of both AVN (Chen et al., 2020) and ours (SAAVN) decline. However, our method has a relatively small reduction in navigation capability and speed compared with AVN, showing better robustness and navigation capabilities.
Independent learning framework does not converge in training. We have also designed a learning framework where the agent and the sound attacker learn independently but not convergent. Our SAAVN employs the multi-agent learning mechanism to train the two policies simultaneously, whose benefit in convergence is empirically demonstrated by Fig. 8.

heading: 5 CONCLUSION
text: This paper proposes a game where an agent competes with a sound attacker in an acoustical intervention environment. We have designed various games of different complexity levels by changing the attack policy regarding the position, sound volume, and sound category. Interestingly, we find that the policy of an agent trained in acoustically complex environments can still perform promisingly in acoustically simple settings, but not vice versa. This observation necessitates our contribution in bridging the gap between audio-visual navigation research and its real-world applications. A complete set of ablation studies is also carried out to verify the optimal choice of our model design and training algorithm. However, the limitation of our work is that we only assume one sound attacker and have not studied the scenario with two and more attackers. Another limitation of our work is that our current evaluations are conducted in virtual environments. It will make better sense to assess our method on practical cases like a robot navigating in a real house. The above two will be left for future exploration. Since our research is conducted on a simulation platform, it is unlikely to cause a negative social impact in the foreseeable future.

heading: 6 REPRODUCIBILITY
text: Our code is available at : appx. L and https://github.com/yyf17/SAAVN/tree/main. The algorithm parameters are detailed in appx. G.
Please refer to appx. A for more details about the acoustically clean or simple environment and acoustically complex environment. For more detailed information about the Fourier transform properties, see appx. B.
The dataset Replica and Matterport3D is used in the experiment. For basic information about the dataset, you can refer to appx. L for more details. Please refer to https://github.com/ yyf17/SAAVN/blob/main/dataset.md for the detailed steps of downloading and processing the dataset.

heading: 7 ETHICS STATEMENT
text: The research in this paper does NOT involve any human subject, and our dataset is not related to any issue of privacy and can be used publicly. All authors of this paper follow the ICLR Code of Ethics (https://iclr.cc/public/CodeOfEthics).

heading: ACKNOWLEDGEMENT
text: The following projects jointly supported this work: the Sino-German Collaborative Research Project Crossmodal Learning (NSFC 62061136001/DFG TRR169); Beijing Science and Technology Plan Project (No.Z191100008019008); the National Natural Science Foundation of China (No.62006137); Natural Science Project of Scientific Research Plan of Colleges and Universities in Xinjiang (No.XJEDU2021Y003).

heading: A DEFINITIONS.
text: Definition 2 Acoustically clean or simple environment. The acoustically clean or simple environment is described as follows: (1) The number of target sound sources is one. (2) The position of the target sound source is fixed in an episode of a scene. (3) The volume of the target sound source is the same in all episodes of all scenes, and there is no change.
Definition 3 The acoustically complex environment. The acoustically complex environment referred to in this study is defined as follows: (1) There is one non-target sound source in the scene. (2) The position of the non-target sound source is uncertain, which means that the position of the non-target sound source in the scene is arbitrary. (3) The volume of the non-target sound source is uncertain. It is only known that the maximum volume of the non-target sound is equal to the maximum volume of the target non-target sound source, but the specific volume is uncertain. (4) The sound category of the non-target sound source is uncertain. Only the set to which the non-target sound category belongs is the same as the set to which the target sound source category belongs. (5)The attacker has no physical entity. It is like an invisible ghost. It has no shape, no volume, and no mass. When the agent runs in front of it, it will not block its movement or collide with the agent. This assumption is to simplify the model. These non-target sounds are a major obstacle to audio-visual embodied navigation, which greatly increases the search time.
Properties of non-target sound sources The non-target sound sources have their characteristics: (1) Although non-target and target sounds belong to the same spectrum range, these non-target sounds are not suitable for Gaussian use. Distribution is used for modeling, and noise is suitable for modeling with Gaussian distribution. It is difficult to model both the non-target low voice and the target’s natural gas alarm sound. (2) The existence of these non-target sounds is a significant obstacle to audio-visual embodied navigation, which significantly increases the search time. (3) These non-target sounds may not move like the target, or they may explore around the scene. Our work is based on the scope of the above definition.

heading: B PROPERTIES OF FOURIER TRANSFORM.
text: Property 2 The following three properties of the Fourier transform need to be used when proving the Theorem 1. 1. Linearity : The Fourier transform of sum of two or more functions is the sum of the Fourier transforms of the functions. F(a+ b) = F(a) +F(b). If we multiply a function by a constant, the Fourier transform of the resultant function is multiplied by the same constant. F(k · a) = k · F(a). 2. Convolution : The Fourier transform of a convolution of two functions is the point-wise product of their respective Fourier transforms. F(f ∗ g) = F(f) · F(g). 3. Dirac delta function : Fourier transform of Dirac delta function is 12π . So F(ψ) = 1 2π .

heading: C ENVIRONMENT.
text: SoundSpaces. SoundSpaces uses the Habitat simulator with the publicly available Replica and Matterport3D environments and the public SoundSpaces audio simulation. The 18 replica environments are grids constructed based on accurate scans of apartments, offices, hotels, and rooms. The 85 Matterport3D environments are real homes and other indoor environments with 3D grids and image scanning. Use SoundSpaces’ room impulse response (RIR) to place the audio source and environmental attackers in a 3D environment, and then simulate realistic sound at each position in the scene, where the spatial resolution of the copy is 0.5m, and the spatial resolution of Matterport3D is 1m. The robot can walk in space while receiving real-time egocentric visual and audio observations.
Rewards. According to the classic navigation rewards, if the robot successfully reaches the target and executes the Stop action, the environment will reward it with +10, plus an additional bonus of 0.25 to reduce the Manhattan distance from the robot to the target. Furthermore, increase the equivalent penalty for this target. Finally, we impose a time penalty of −0.01 on each action performed to encourage efficiency.
Action space. SoundSpaces maintains a navigability graph of the environment (unknown to the agent). The agent can only move from one node to another if an edge is connecting them and the agent is facing that direction. The action space of the agent are navigation motions: Aω = {MoveForward,TurnLeft,TurnRight, Stop}. An environment attacker embodied in the environment must take actions from a hybrid action space Aν . The hybrid action space are Cartesian product of navigation motions spaceAν,position,volume of sound spaceAν,volume and category of sound spaceAν,category: Aν = Aν,position×Aν,volume×Aν,category ,whereAν,position = Aω is navigation motion space, Aν,volume = {0.0, 0.1, 0.2, . . . , 1.0} is discrete action space with |Aν,volume| = 11 , Aν,category = {′telephone′,′ person′, . . .} is discrete action space with |Aν,category| = 101 respectively, We experiment with 101 everyday sounds.We use 101 natural sounds without duplication in SoundSpaces, covering various categories: birds, air conditioners, doorbells, door openings, music, computer beeps, fans, talkers, phones, and etc.
D IMPLEMENTATION DETAILS.
In the following, we provide details of our reinforcement learning (RL) for navigation tasks. The environment attacker embodied in an environment must take actions from an action space Aν to make a change of the acoustically simple environment of SoundSpaces. An agent embodied in an environment must take actions from an action space Aω to accomplish an end goal. At every time step, t = {0, 1, 2, . . . , T − 1} the environment is in some state, but the environment attacker and the agent obtain only a partial observation of it in the form of st. Here T is a maximal
time horizon, corresponding to 500 actions in a scene of a scene for our task. The observation st is a combination of the audio, and visual inputs. The environment attacker develops a hybrid policy πνt,θ = π ν,position t,θ × π ν,volume t,θ × π ν,category t,θ , where πν,positiont,θ : Aν,position → R3 , π ν,volume t,θ : Aν,volume → R11 , π ν,category t,θ : Aν,category → R101 respectively, and where πνt,θ(at | st, ht−1) is the probability that the environment attacker chooses to take action aν ∈ Aν at time t given the current observation st and aggregated past states ht−1 . Using information about the previous time steps ht−1 and current observation st, the agent develops a policy πωθ : Aω → R4 , where πωt,θ(at | st, ht−1) is the probability that the agent chooses to take action aω ∈ Aω at time t given the current observation st and aggregated past states ht−1 . After the environment attacker and the agent acts, the environment goes into a new state st+1 and the agent and attacker receives individual rewards rωt ∈ R and rνt ∈ R respectively. The agent optimizes its return, i.e. the expected discounted, cumulative rewards Gωγ,t =∑T−1
t=0 γ trωt . The environment attacker also optimizes its return, i.e. the expected discounted, cumulative rewards Gνγ,t = ∑T−1 t=0 γ trνt ,where γ ∈ [0, 1] is the discount factor to modulate the emphasis on recent or long term rewards. The value function V ωt,θ(st, ht−1) and V ν t,θ(st, ht−1) is the expected return for agent and attacker respectively. We optimize the particular reinforcement learning objective directly following from Proximal Policy Optimization(PPO). We refer the readers to PPO for additional details on optimization.
We train our model with Adam with a learning rate of 2.5× 10−4. The auditory and visual encoder output are 512 and 512, respectively. We use a one-layer bidirectional GRU with 512 hidden units that take [It, bt] as input. We use an entropy loss on the policy distribution with a coefficient of 0.01. We train the network for 30M agent steps on Replica and 60M on Matterport3D, which amounts to 150 and 320 GPU hours, respectively.
Following SoundSpace, we first compute the Short-Time Fourier Transform (STFT) with a hop length of 160 samples and a windowed signal length of 512 samples, which corresponds to a physical duration of 12 and 32 milliseconds at a sample rate of 44100Hz (Replica) and 16000Hz (Matterport). STFT gives a 257 × 257 and a 257 × 101 complex-valued matrix, respectively, for a one-second audio clip; we take its magnitude, downsample both axes by a factor of 4, and take the logarithm. Finally, we stack the left and right audio channel matrices to obtain a 65× 65× 2 and a 65× 26× 2 tensor.

heading: E SOUNDSPACES DATASET IN DETAILS.
text: We used the SoundSpaces dataset. Table 3 summarizes SoundSpaces dataset,which includes audio renderings for the Replica and Matterport3D datasets. Each episode consists of a tuple: ⟨scene, agent start location, agent start rotation, goal location, audio waveform⟩. Episodes were generated by choosing a scene and a random start and goal location.

heading: F METRICS IN DETAILS.
text: Next, we narrate the navigation metrics used in Section 4 of the body paper.
1. Success weighted by Path Length (SPL): weighs the successful episodes with the ratio of the shortest path li to the executed path pi, SPL = 1N ∑N i=1 Si li max(pi,li)
, where Si is binary indicator of success in episode i, and N is the number of episodes.
2. Rmean: stands for average episode reward of agent. See Appendix C for more details on reward.
3. Soft Success weighted by Path Length (SSPL, or SoftSPL): Similar to SPL with a relaxed softsuccess criteria. SSPL = 1N ∑N i=1 max(0, 1− dai di ) limax(pi,li) , where d a i is the distance from the
agent’s current position to the goal when episode i is finished, di is the distance from the agent’s start position to the goal in the episode i, li is the shortest path, pi is the executed path.
4. Success Rate (SR): The fraction of completed episodes, the agent reaches the goal within the time limit of 500 steps and selects the stop action precisely at the goal location, SR = 1N ∑N i=1 Si.
5. Average Distance To Goal (DTG): The agent’s average distance to the goal when episodes are finished, DTG = 1N ∑N i=1 d a i , where d a i is the distance from the agent’s current position to the
goal when episode i is finished. 6. Normalized average Distance To Goal (NDTG) : NDTG = 1N ∑N i=1 dai di
, where dai is the distance from the agent’s current position to the goal when episode i is finished, di is the distance from the agent’s start position to the goal in the episode i.

heading: G ALGORITHM PARAMETERS.
text: The parameters used in our model are shown in Table 4.

heading: H EXPERIMENTS RESULTS IN DETAIL.
text: It can be concluded from the main paper that AVN’s navigation capabilities are better than SA-MDP. So we focus on comparing the navigation capabilities of AVN and our model SAAVN on dataset Matterport3D. We select the model variant SAAVN: PVC for experimental comparison to prove the effectiveness of our model. Later we will show navigation results on dataset Replica.
Navigation results on dataset Matterport3D. Table 5 shows the comparative experiments of different models on the dataset Matterport3D in a clean environment. It can be seen from Table 5 that our model is the best under all metrics in a simple or clean environment.
The experimental comparison of different models on the dataset Matterport3D in an acoustically complex environment is shown in Table 6. It can be seen from Table 6 that our model is the best under all metrics in all the acoustically complex environments. Complex env in the Table 7 stands for the acoustically complex environments which includes (PV C)random and PV C. As can be seen from Table 7, for the two acoustically complex environments (PV C)random and PV C, our model wins under both SPL and Rmean metrics. From the above Table 5, Table 6 and Table 7, it can be concluded that our model achieves better navigation capabilities than AVN in all environments on dataset Matterport3D under all metrics.
Navigation results on dataset Replica. The results of comparative experiments in the clean environment of dataset Replica are shown in Table 8. It can be seen from Table 8 that for the dataset Replica, in a simple or clean environment, the performance of our model variants SAAVN:P, V=0.9, C=person6, and SAAVN:P, V=0.1, C=person6 under all metrics are better than AVN’s, which fully shows that our model has more robust navigation capabilities in an acoustically simple environment.
The comparative experiments in the acoustically complex environments on the dataset Replica are shown in Table 9. Complex Env in the Table 9 stands for acoustically complex environments. It can be seen from Table 9 that our various model variants achieve better navigation capabilities than AVN respectively in different acoustically complex environments on dataset Replica under all metrics.
The data in Table 8 and Table 9 can be visualized with the histogram of Figure 9. Figure 9 shows the comparative experimental results of dataset Replica in all environments. The two bars on the left of each box in the Fig. 9 are comparisons of the clean environments, and the two bars on the right are comparisons of the acoustically complex environments. It can be seen from the histogram that our model performs better than AVN in both simple clean environment and acoustically complex environment.
♡:Model variant SAAVN:P,V=0.1,C=person6 ranked second in performance on metrics SPL and SSPL, and ranked first in performance on metrics SR, Rmean, DTG and NDTG. ♣: Model variant SAAVN:P, V=0.9, C=person6 ranks first in performance on metrics SPL and SSPL, ranked second in performance on metrics Rmean, DTG and NDTG, and ranked third in performance on the metric SR. Based on the above factors, we believe that model variant SAAVN:P, V=0.1, C=person6 is the best. So in Table 8 and the main paper, we mark model variant SAAVN:P, V=0.1, C=person6 as the best model.

heading: I TRAJECTORY EXAMPLES IN DETAILS.
text: Trajectory comparisons on dataset Replica. Figure 10 shows the test episodes for our SAAVN model on dataset Replica. The environment of each row in Fig. 10 is consistent, and the model of each column is too. SA-MDP failed to complete the task successfully in all two environments. AVN can complete the job in a clean environment but fail to reach the target in an acoustically complex environment. SAAVN completed the tasks in all two environments. What is more, the navigation track of SAAVN in an acoustically simple environment is shorter than that of AVN. These fully demonstrate the navigation capabilities of SAAVN.
Trajectory comparisons on dataset Matterport3D. Figure 11 and Figure 12 shows the test episodes for our SAAVN model on dataset Matterport3D. The environment of each row in the figures is consistent, and the model of each column is too. AVN can complete the task in a clean environment but fails in an acoustically complex environment. SAAVN completed the tasks in all two environments. What is more, the navigation track of SAAVN in an acoustically simple environment is shorter than that of AVN. These fully demonstrate the navigation capabilities of SAAVN.
J INDEPENDENT LEARNING FRAMEWORK DOES NOT CONVERGE IN TRAINING.
Conventional adversarial approaches usually train the agent’s policy and the attacker’s policy alternately and independently. Instead, our paper employs the multi-agent learning mechanism to train the two policies simultaneously, whose benefit in convergence is empirically demonstrated by Figure 8 (by comparing our SAAVN with IDL). We have also designed a learning framework in the algorithm design process in which the agent and the sound attacker learn independently. It can be seen from Figure 8 that the framework of independent learning(IDL) by the agent and the sound attacker is not convergent.

heading: K MORE DISCUSSION ABOUT OUR MODEL SAAVN.
text: Although our work primarily focuses on intervention on the sound modality. We have some discussion on the robustness of SAAVN against the visual modality intervention, the robustness of SAAVN on robot blindness, etc. For more details, please see below.
K.1 ROBUSTNESS OF SAAVN AGAINST THE VISUAL MODALITY.
Our work primarily focuses on intervention on the sound modality. What is robustness against the visual modality? We add gaussian noise with different std and zero mean to the visual observation (depth image) and evaluation the performance of our model SAAVN on Matterport3D. The result is demonstrated in Table 10 show that When the intensity of intervention increases, performance decreases. However, the decline is slow. This phenomenon shows that our model SAAVN shows better robustness under visual modal intervention.
K.2 ROBUSTNESS OF SAAVN AGAINST BLINDNESS.
When the visual sensor is wholly removed, that is to say, and the robot is blind. What is the performance of our model and AVN? We design and do blindness experiments on Matterport3D. The experimental results in Table 11 show that the performance of SPL and Rmean of model AVN and model SAAVN has decreased. ∆SPL stands for the deterioration amount under metric SPL for a specific model. The performance value calculates ∆SPL with removing visual sensor intervention minus the commission value under metric SPL without intervention under metric SPL. ∆Rmean stands for the deterioration amount under metric Rmean for a specific model. ∆Rmean is calculated by the performance value with removing visual sensor intervention minus the commission value without intervention for a specific model under metric Rmean. The performance degradation of SPL and Rmean of model AVN is greater than that of model SAAVN. It shows that the SAAVN model is more robust than the model AVN when faced with the complete removal of the vision sensor.
K.3 PERFORMANCE AFFECT BY SLIDING AND SKIPPING MODES OF aν,VOL .
The action aν,vol can take by sliding and skipping modes. Suppose the current action aν,vol is 0.5. What is the next step action ? If using the sliding mode, the next action of aν,vol is to select one from neibour actions ([0.4, 0.5, 0.6]) of current action. If using the skip mode, the next action of aν,vol is to select one from whole action space of aν,vol ([0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]). How does the choice of action mode affect the performance of the model SAAVN? The experimental results on Matterport3D in Table 12 show that the performance of model SAAVN adopts the sliding mode is better than that of the skipping mode for action aν,vol under metrics SPL and Rmean.
K.4 UNSEEN ENVIRONMENTS.
We test our method by assigning the attacker with the sound of a random category that is unseen in training. The following table summarizes the results of AVN and SAAVN. As a reference, we also copy the results from Table 2 that are evaluated under the original ”seen” setting. It suggests that SAAVN still performs desirably, particularly for the environments ”V” and ”C”, whereas AVN yields more significant detriment in these two cases (See Table 13, 14, 15).
K.5 SAAVN: PVC ACHIEVES THE BEST PERFORMANCE.
Here, we further evaluate different methods in the same attack environment, ”PVC,” as follows. As expected, SAAVN: PVC achieves the best performance (See Table 16).
K.6 MULTI-MODAL FUSION ABLATION FOR SOUND ATTACKER AUDIO-VISUAL NAVIGATION.
Is concatenation the best choice for multi-modal fusion? We did not explore which choice is the best for multi-modal fusion in the main paper since this is not the main focus of our paper (all methods share the same fusion setting). However, we do think this is an exciting point. Hence, besides concatenation, we have also conducted another fusion strategy using element-wise multiplication. Specifically, we first obtain the visual and audio embedding vectors of the same size (i.e., 512) and then compute the element-wise multiplication between these two vectors. Table 17 reports the performance of SAAVN: PVC in the environment PVC under these two different fusion scenarios.
Interestingly, the new fusion strategy is better than the original concatenation. Upon the observation here, we believe that our proposed method allows further extendibility and can facilitate various following studies.

heading: L CODES.
text: Codes are in the folder named code of the supplementary materials. The usage and main introduction of the code are in the readme.MD.
M VIDEO DEMONSTRATIONS ON DATASETS REPLICA AND MATTERPORT3D.
Video demonstrations of navigation trajectory in an episode of different models in the clean environment and acoustically complex environments are in the folder named demoVideo of the supplementary materials. There are two subfolders under this folder named DemonstrationOnDatasetReplica and DemonstrationOnDatasetMatterport3D, used to store trajectory videos on the dataset Replica the dataset Matterport3D, respectively. Each MP4 file is named according to the format: xxx in yyy env zzz, where xxx represents the name of the model, xxx ∈ {AV N,SAAV N}, yyy represents the type of environment, yyy ∈ {simple, complex}, and zzz represents the name of the datasets, zzz ∈ {Replica,Matterport3D}.

heading: N BROADER IMPACT.
text: Our research will help service robots provide better navigation services in home life scenes and indoor office scenes. The setting of our research work is based on the complex acoustic environment. The gap between this research setting and the actual application setting is small, conducive to applying the research results to real application scenarios. The limitation of our work is that we only assume one sound attacker and have not studied the scenario with two and more attackers, which will be left for future exploration.

Review summaries:
The paper proposes an interference-robust training method for audio-visual navigation. Unlike existing approaches that focus on clean environments, the system is trained in simulated acoustically complex environments. A single-source adversarial attacker is introduced, which determines position, noise type and volume that would make the agent to suffer most. This is claimed to improve robustness to random attacks in AVN. Results are compared to a small selection of previous works.

This paper addressed the audio-visual navigation task in the environment of sound attacker. The authors formulated the problem as zero-sum game between sound attacker and the agent, and provide a reasonable proof to their formulation and solution. They showed better results than existing works which don't considered this kind of attack (but, I am not sure for SA-MDP). 

The authors address the audio-visual navigation task in this paper. Previous works usually assume that the interacted environment is clean containing solely the target sound. Different from them, this work explores an acoustically complex environment in which, besides the target sound, there exists a sound attacker playing a zero-sum game with the agent. Specifically, the attacker can move and change the volume and sound category to fool the agent while the agent tries to defend against the attack and navigate to the goal under the intervention. Experiments on Replica and Matterport3D can verify the effectiveness and the robustness of proposed model.