Sound Adversarial Audio-Visual Navigation

1 INTRODUCTION . Audio-visual navigation , as currently a vital task for embodied vision ( Gordon et al. , 2018 ; Lohmann et al. , 2020 ; Nagarajan & Grauman , 2020 ) , requires the agent to find a sound target in a realistic and unmapped 3D environment by exploring with egocentric audio-visual observations ( Chen et al. , 2019 ; Gupta et al. , 2017 ; Chaplot et al. , 2020 ) . Inspired by the simultaneous usage of eyes and ears in human exploration ( Wilcox et al. , 2007 ; Flom & Bahrick , 2007 ) , audio-visual association benefits the learning of the agent ( Dean et al. , 2020 ) . A recent work Look , Listen , and Act ( LLA ) has proposed a three-step navigation solution of perception , inference , and decision-making ( Gan et al. , 2020 ) . SoundSpaces is the first work to establish an audio-visual embodied navigation simulation platform equipped with the proposed Audio-Visual embodied Navigation ( AVN ) baseline that resorts to reinforcement learning ( Chen et al. , 2020 ) . In response to the long-term exploration problem that is caused by the large layout of the 3D scene and the long distance to the target place , Audio-Visual Waypoint Navigation ( AV-WaN ) proposes an audio-visual navigation algorithm by setting waypoints as sub-goals to facilitate sound source discovering ( Chen et al. , 2021b ) . Besides , Semantic Audio-Visual navigation ( SAVi ) develops a navigation algorithm in a scene where the target sound is not periodic and has a variable length ; that is , it may stop during the navigation process ( Chen et al. , 2021a ) . Despite the fruitful progress in audio-visual navigation , existing works assume an acoustically simple or clean environment , meaning there is no other sound but the source itself . Nevertheless , this ∗Corresponding author : Fuchun Sun . assumption is hardly the case in real life . For example , a kettle in the kitchen beeps to tell the robot that the water is boiling , and the robot in the living room needs to navigate to the kitchen and turn off the stove ; while in the living room , two children are playing a game , chuckling loudly from time to time . Such an example poses a crucial challenge on current techniques : can an agent still find its way to the destination without being distracted by all non-target sounds around the agent ? Intuitively , the answer is no if the agent has not been trained in acoustically complex environments as in the example listed above . Although the answer is no , this ability is what we expect the agent to possess in real life . In light of these limitations , we propose first to construct such an acoustically complex environment . In this environment , we add a sound attacker to intervene . This sound attacker can move and change the volume and type of the sound at each time step . In particular , the objective of the sound attacker is to make the agent frustrated by creating a distraction . In contrast , the agent decides how to move at every time step , tries to dodge the sound attack , and explores for the sound target well under the sound attack , as illustrated in Fig . 1 . The competition between the attacker and the agent can be modeled as a zero-sum two-player game . Notably , this is not a fair game and is more biased towards the agent for two reasons . First , the sound attack is just single-modal and will not intervene in any visual information obtained by the agent . Second , as will be specified in our methodology , the sound volume of the attacker is bounded via a relative ratio ( less than 1 ) of the sound target . We anticipate constraining the attacker ’ s power and encouraging it to intervene but not defeat the agent with these two considerations . With such a design , we can improve the agent ’ s robustness between the agent and the sound attacker during the game . On the other hand , our environment is more demanding than reality since there are few attackers in our lives . Instead , most behaviors , such as someone walking and chatting past the robot , are not deliberately embarrassing the robot but just a distraction to the robot , exhibiting weaker intervention strength than our adversarial setting . Even so , our experiments reveal that an agent trained in a worst-case setting can perform promisingly when the environment is acoustically clean or contains a natural sound intervenor using a random policy . On the contrary , the agent trained in a clean environment becomes disabled in an acoustically complex environment . Our training algorithm is built upon the architecture by ( Chen et al. , 2020 ) , with a novel decisionmaking branch for the attacker . Training two agents separately ( Tampuu et al. , 2017 ) leads to divergence . Hence we propose a joint Actor-Critic ( AC ) training framework . We define the policies for the attacker based on three types of information : position , sound volume , and sound category . Exciting discoveries from experiments demonstrate that the joint training converges promisingly in contrast to the independent training counterpart . This work is the first audio-visual navigation method with a sound attacker to the best of our knowledge . To sum up , our contributions are as follows . • We construct a sound attacker to intervene environment for audio-visual navigation that aims to improve the agent ’ s robustness . In contrast to the environment used by prior experiments ( Chen et al. , 2020 ) , our setting better simulates the practical case in which there exist other moving intervenor sounds . • We develop a joint training paradigm for the agent and the attacker . Moreover , we have justified the effectiveness of this paradigm , both theoretically and empirically . • Experiments on two real-world 3D scenes , Replica ( Straub et al. , 2019 ) and Matterport3D ( Chang et al. , 2017 ) validate the effectiveness and robustness of the agent trained under our designed environment when transferred to various cases , including the clean environment and the one that contains sound attacker with random policy . 2 RELATED WORK . We introduce the research related to our work , including audio-visual navigation , adversarial training in RL , and multi-agent learning . Audio-visual embodied navigation . This task demands a robot equipped with a camera and a microphone to interact with the environment and navigate to the source sound . Existing algorithms towards this task can be divided into two categories according to whether topological maps are constructed or not . For the first category ( Gan et al. , 2020 ; Chen et al. , 2021b ) , LLA ( Gan et al. , 2020 ) plans the robot ’ s navigation strategy based on graph-based shortest path planning . Following LLA , AV-WaN ( Chen et al. , 2021b ) combines dynamically setting waypoints with the shortest path algorithm to solve the long-period audio-visual navigation problem . The second category of methods works in the absence of a topological map ( Chen et al. , 2021a ; 2020 ) . In particular , SAVi ( Chen et al. , 2021a ) aims to solve the audio-visual navigation problem with temporary sound source by introducing the category information of the source sound and pre-training the displacement from the robot to the sound source ; AVN ( Chen et al. , 2020 ) constructs the first audio-visual embodied navigation simulation platform—SoundSpaces and makes use of Reinforcement Learning ( RL ) for the training of the navigation policy . As presented in the Introduction , all environments used previously ( including SoundSpaces ) assume clean sound sources , which is hardly the case in real and noisy life . By contrast , we build an acoustically complex environment that allows a sound attacker to move and change the volume and category of sound at each time step in an episode . In this environment , we train the navigation policy of the agent under the sound attack , which delivers a more robust solution in real applications . Adversarial attacks and adversarial training in RL . In general , adversarial attacks ( Tian & Xu , 2021 ) /training in RL are divided into two classes : learning to attack ( Huang et al. , 2017 ; Kos & Song , 2017 ; Lin et al. , 2017 ; Pattanaik et al. , 2018 ) and learning to defence ( Pinto et al. , 2017 ; Gleave et al. , 2020 ; Zhang et al. , 2020 ) that targets on learn a robustness policy by state adversarial through external forces or sensor noise . Our paper falls into the second class . A close work to our method is State Adversarial MDP ( SA-MDP ) ( Zhang et al. , 2020 ) that leverages sensor noise in vision to improve the robustness of the algorithm . The main difference between our method and SA-MDP is that our method precisely permits a sound attacker to move in the room and employs the resulting sound as a distractor , while in SA-MDP , the adversarial state is created arbitrarily and thus not necessarily fits the actual scene . Besides , SA-MDP initially copes with the visual modality ; hence , we have changed SA-MDP for the attack of the sound modality for the comparison with our method ( See § 4 ) . Multi-agent learning . We design two frameworks similar to independent learning ( Tampuu et al. , 2017 ) and multi-agent learning ( Sunehag et al. , 2018 ) . However , the framework of independent learning does not converge ( See § 4 ) . We formulate our learning algorithm as a two-player game employed the property of a centralized critic with decentralized actors ( Wang et al. , 2021 ) to guarantee the training convergence of our method in theory ( See Theorem 1 ) . 3 SOUND ADVERSARIAL AUDIO-VISUAL NAVIGATION . 3.1 PROBLEM DEFINITION . Problem modeling of ours . We model the agent as playing against an attacker in a two-player Markov game ( Simon , 2016 ) . We denote the agent and attacker by superscript ω and ν , respectively . The gameM = ( S , ( Aω , Aν ) , P , ( Rω , Rν ) ) consists of state set S , action sets Aω and Aν , and a joint state transition functionP : S×Aω×Aν → S . The reward functionRω : S×Aω×Aν×S → R for agent and Rν : S × Aω × Aν × S → R for attacker respectively depends on the current state , next state and both the agent ’ s and the attacker ’ s actions . Each player wishes to maximize their discounted sum of rewards , where Rω = r , Rν = −r . r is the reward given by the environment at every time step in an episode . Our problem is modeled as following ( See Fig . 2 ( c ) ) : π⋆ = ( π⋆ , ω , π⋆ , ν ) = argmax πω∈Πω { argmin πν∈Πν { G ( πω , πν , r ) } } ( 1 ) Inspired by value decomposition networks ( Sunehag et al. , 2018 ) and QMIX ( Rashid et al. , 2018 ) , we design G ( πω , πν , r ) as Equation ( 2 ) , where G ( πω , r ) and G ( πν , r ) are expected discounted , cumulative rewards of the agent and the attacker respectively . G ( πω , πν , r ) = [ G ( πω , r ) , G ( πν , r ) ] ( 2 ) SA-MDP . As a reference , we introduce the previous adversarial MDP proposed by ( Zhang et al. , 2021 ) as well . In SA-MDP ( See Fig . 2 ( b ) ) , we optimize δadv : = argmax δ : ∥δ∥≤ϵ DKL [ π ω ( a|s ) | πω ( a|s+ δ ) ] . Intuitively , the state of SA-MDP is on the ball with radius ϵ .

Summary: This paper addressed the audio-visual navigation task in the environment of sound attacker. The authors formulated the problem as zero-sum game between sound attacker and the agent, and provide a reasonable proof to their formulation and solution. They showed better results than existing works which don't considered this kind of attack (but, I am not sure for SA-MDP).