Sound Adversarial Audio-Visual Navigation

1 INTRODUCTION . Audio-visual navigation , as currently a vital task for embodied vision ( Gordon et al. , 2018 ; Lohmann et al. , 2020 ; Nagarajan & Grauman , 2020 ) , requires the agent to find a sound target in a realistic and unmapped 3D environment by exploring with egocentric audio-visual observations ( Chen et al. , 2019 ; Gupta et al. , 2017 ; Chaplot et al. , 2020 ) . Inspired by the simultaneous usage of eyes and ears in human exploration ( Wilcox et al. , 2007 ; Flom & Bahrick , 2007 ) , audio-visual association benefits the learning of the agent ( Dean et al. , 2020 ) . A recent work Look , Listen , and Act ( LLA ) has proposed a three-step navigation solution of perception , inference , and decision-making ( Gan et al. , 2020 ) . SoundSpaces is the first work to establish an audio-visual embodied navigation simulation platform equipped with the proposed Audio-Visual embodied Navigation ( AVN ) baseline that resorts to reinforcement learning ( Chen et al. , 2020 ) . In response to the long-term exploration problem that is caused by the large layout of the 3D scene and the long distance to the target place , Audio-Visual Waypoint Navigation ( AV-WaN ) proposes an audio-visual navigation algorithm by setting waypoints as sub-goals to facilitate sound source discovering ( Chen et al. , 2021b ) . Besides , Semantic Audio-Visual navigation ( SAVi ) develops a navigation algorithm in a scene where the target sound is not periodic and has a variable length ; that is , it may stop during the navigation process ( Chen et al. , 2021a ) . Despite the fruitful progress in audio-visual navigation , existing works assume an acoustically simple or clean environment , meaning there is no other sound but the source itself . Nevertheless , this ∗Corresponding author : Fuchun Sun . assumption is hardly the case in real life . For example , a kettle in the kitchen beeps to tell the robot that the water is boiling , and the robot in the living room needs to navigate to the kitchen and turn off the stove ; while in the living room , two children are playing a game , chuckling loudly from time to time . Such an example poses a crucial challenge on current techniques : can an agent still find its way to the destination without being distracted by all non-target sounds around the agent ? Intuitively , the answer is no if the agent has not been trained in acoustically complex environments as in the example listed above . Although the answer is no , this ability is what we expect the agent to possess in real life . In light of these limitations , we propose first to construct such an acoustically complex environment . In this environment , we add a sound attacker to intervene . This sound attacker can move and change the volume and type of the sound at each time step . In particular , the objective of the sound attacker is to make the agent frustrated by creating a distraction . In contrast , the agent decides how to move at every time step , tries to dodge the sound attack , and explores for the sound target well under the sound attack , as illustrated in Fig . 1 . The competition between the attacker and the agent can be modeled as a zero-sum two-player game . Notably , this is not a fair game and is more biased towards the agent for two reasons . First , the sound attack is just single-modal and will not intervene in any visual information obtained by the agent . Second , as will be specified in our methodology , the sound volume of the attacker is bounded via a relative ratio ( less than 1 ) of the sound target . We anticipate constraining the attacker ’ s power and encouraging it to intervene but not defeat the agent with these two considerations . With such a design , we can improve the agent ’ s robustness between the agent and the sound attacker during the game . On the other hand , our environment is more demanding than reality since there are few attackers in our lives . Instead , most behaviors , such as someone walking and chatting past the robot , are not deliberately embarrassing the robot but just a distraction to the robot , exhibiting weaker intervention strength than our adversarial setting . Even so , our experiments reveal that an agent trained in a worst-case setting can perform promisingly when the environment is acoustically clean or contains a natural sound intervenor using a random policy . On the contrary , the agent trained in a clean environment becomes disabled in an acoustically complex environment . Our training algorithm is built upon the architecture by ( Chen et al. , 2020 ) , with a novel decisionmaking branch for the attacker . Training two agents separately ( Tampuu et al. , 2017 ) leads to divergence . Hence we propose a joint Actor-Critic ( AC ) training framework . We define the policies for the attacker based on three types of information : position , sound volume , and sound category . Exciting discoveries from experiments demonstrate that the joint training converges promisingly in contrast to the independent training counterpart . This work is the first audio-visual navigation method with a sound attacker to the best of our knowledge . To sum up , our contributions are as follows . • We construct a sound attacker to intervene environment for audio-visual navigation that aims to improve the agent ’ s robustness . In contrast to the environment used by prior experiments ( Chen et al. , 2020 ) , our setting better simulates the practical case in which there exist other moving intervenor sounds . • We develop a joint training paradigm for the agent and the attacker . Moreover , we have justified the effectiveness of this paradigm , both theoretically and empirically . • Experiments on two real-world 3D scenes , Replica ( Straub et al. , 2019 ) and Matterport3D ( Chang et al. , 2017 ) validate the effectiveness and robustness of the agent trained under our designed environment when transferred to various cases , including the clean environment and the one that contains sound attacker with random policy . 2 RELATED WORK . We introduce the research related to our work , including audio-visual navigation , adversarial training in RL , and multi-agent learning . Audio-visual embodied navigation . This task demands a robot equipped with a camera and a microphone to interact with the environment and navigate to the source sound . Existing algorithms towards this task can be divided into two categories according to whether topological maps are constructed or not . For the first category ( Gan et al. , 2020 ; Chen et al. , 2021b ) , LLA ( Gan et al. , 2020 ) plans the robot ’ s navigation strategy based on graph-based shortest path planning . Following LLA , AV-WaN ( Chen et al. , 2021b ) combines dynamically setting waypoints with the shortest path algorithm to solve the long-period audio-visual navigation problem . The second category of methods works in the absence of a topological map ( Chen et al. , 2021a ; 2020 ) . In particular , SAVi ( Chen et al. , 2021a ) aims to solve the audio-visual navigation problem with temporary sound source by introducing the category information of the source sound and pre-training the displacement from the robot to the sound source ; AVN ( Chen et al. , 2020 ) constructs the first audio-visual embodied navigation simulation platform—SoundSpaces and makes use of Reinforcement Learning ( RL ) for the training of the navigation policy . As presented in the Introduction , all environments used previously ( including SoundSpaces ) assume clean sound sources , which is hardly the case in real and noisy life . By contrast , we build an acoustically complex environment that allows a sound attacker to move and change the volume and category of sound at each time step in an episode . In this environment , we train the navigation policy of the agent under the sound attack , which delivers a more robust solution in real applications . Adversarial attacks and adversarial training in RL . In general , adversarial attacks ( Tian & Xu , 2021 ) /training in RL are divided into two classes : learning to attack ( Huang et al. , 2017 ; Kos & Song , 2017 ; Lin et al. , 2017 ; Pattanaik et al. , 2018 ) and learning to defence ( Pinto et al. , 2017 ; Gleave et al. , 2020 ; Zhang et al. , 2020 ) that targets on learn a robustness policy by state adversarial through external forces or sensor noise . Our paper falls into the second class . A close work to our method is State Adversarial MDP ( SA-MDP ) ( Zhang et al. , 2020 ) that leverages sensor noise in vision to improve the robustness of the algorithm . The main difference between our method and SA-MDP is that our method precisely permits a sound attacker to move in the room and employs the resulting sound as a distractor , while in SA-MDP , the adversarial state is created arbitrarily and thus not necessarily fits the actual scene . Besides , SA-MDP initially copes with the visual modality ; hence , we have changed SA-MDP for the attack of the sound modality for the comparison with our method ( See § 4 ) . Multi-agent learning . We design two frameworks similar to independent learning ( Tampuu et al. , 2017 ) and multi-agent learning ( Sunehag et al. , 2018 ) . However , the framework of independent learning does not converge ( See § 4 ) . We formulate our learning algorithm as a two-player game employed the property of a centralized critic with decentralized actors ( Wang et al. , 2021 ) to guarantee the training convergence of our method in theory ( See Theorem 1 ) . 3 SOUND ADVERSARIAL AUDIO-VISUAL NAVIGATION . 3.1 PROBLEM DEFINITION . Problem modeling of ours . We model the agent as playing against an attacker in a two-player Markov game ( Simon , 2016 ) . We denote the agent and attacker by superscript ω and ν , respectively . The gameM = ( S , ( Aω , Aν ) , P , ( Rω , Rν ) ) consists of state set S , action sets Aω and Aν , and a joint state transition functionP : S×Aω×Aν → S . The reward functionRω : S×Aω×Aν×S → R for agent and Rν : S × Aω × Aν × S → R for attacker respectively depends on the current state , next state and both the agent ’ s and the attacker ’ s actions . Each player wishes to maximize their discounted sum of rewards , where Rω = r , Rν = −r . r is the reward given by the environment at every time step in an episode . Our problem is modeled as following ( See Fig . 2 ( c ) ) : π⋆ = ( π⋆ , ω , π⋆ , ν ) = argmax πω∈Πω { argmin πν∈Πν { G ( πω , πν , r ) } } ( 1 ) Inspired by value decomposition networks ( Sunehag et al. , 2018 ) and QMIX ( Rashid et al. , 2018 ) , we design G ( πω , πν , r ) as Equation ( 2 ) , where G ( πω , r ) and G ( πν , r ) are expected discounted , cumulative rewards of the agent and the attacker respectively . G ( πω , πν , r ) = [ G ( πω , r ) , G ( πν , r ) ] ( 2 ) SA-MDP . As a reference , we introduce the previous adversarial MDP proposed by ( Zhang et al. , 2021 ) as well . In SA-MDP ( See Fig . 2 ( b ) ) , we optimize δadv : = argmax δ : ∥δ∥≤ϵ DKL [ π ω ( a|s ) | πω ( a|s+ δ ) ] . Intuitively , the state of SA-MDP is on the ball with radius ϵ .

Summary: The paper proposes an interference-robust training method for audio-visual navigation. Unlike existing approaches that focus on clean environments, the system is trained in simulated acoustically complex environments. A single-source adversarial attacker is introduced, which determines position, noise type and volume that would make the agent to suffer most. This is claimed to improve robustness to random attacks in AVN. Results are compared to a small selection of previous works.