AD-VAT: An Asymmetric Dueling mechanism for learning Visual Active Tracking

1 INTRODUCTION . Visual Active Tracking ( VAT ) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations . VAT is demanded in many real-world applications such as autonomous vehicle fleet ( e.g. , a slave-vehicle should follow a master-vehicle ahead ) , service robots and drones ( e.g. , a drone is required to follow a person when recording a video ) . To accomplish the VAT task , one typically needs to perform a sequence of tasks such as recognition , localization , motion prediction , and camera control . However , conventional visual tracking ( Babenko et al. , 2009 ; Ross et al. , 2008 ; Mei & Ling , 2009 ; Hu et al. , 2012 ; Bolme et al. , 2010 ; Kalal et al. , 2012 ) aims to solely propose a 2D bounding box of the target frame by frame , and does not actively take into consideration the control of camera . Thus , compared to the problem of “ passive ” tracking , VAT is more practical and challenging . With the advancement of deep reinforcement learning ( Sutton & Barto , 1998 ; Mnih et al. , 2015 ; Silver et al. , 2016 ; Mnih et al. , 2016 ) , training an end-to-end deep neural network via reinforcement learning for VAT is shown to be feasible ( Luo et al. , 2018 ; Luo et al. , 2019 ) . The authors learn a policy that maps raw-pixel observation to control signal straightly with a Conv-LSTM network . Such an end-to-end approach could save the effort of tuning an extra camera controller . Meanwhile , it also outperforms the conventional methods where the passive tracker is equipped with a hand-engineered camera controller . However , the performance of the deep reinforcement learning based tracker is still limited by the training methods . Due to the “ trial-and-error ” nature of reinforcement learning , it is infeasible to directly train the tracker in the real world . Alternatively , virtual environments are always utilized to generate sufficient data for training without tedious human labeling . Nevertheless , to deploy the trained tracker in the real world , one has to overcome the virtual-to-real gap . One solution can be building numbers of high-fidelity environments ( Zhu et al. , 2017 ) . However , it is expensive and tedious to build such environments for VAT . Both the visual rendering ( illumination , texture , etc . ) and the physical properties should be carefully designed to emulate the real world . Suppose we carry out VAT where the target is a pedestrian . To build the environment , one has to not only model the human ’ s appearance , but also design physical rules and the pedestrian ’ s trajectory so that it moves naturally like a human beings . Recently , ( Luo et al. , 2018 ) tried to overcome the virtual-to-real gap by applying the so-called environment augmentation technique . They diversify the visual appearance by changing the placement of the background objects and by flipping left-right the screen frame . However , they neglect another important factor , that is , the motion of the target for VAT task . Intuitively , the complexity and diversity of the target motion in training will impact the generalization of the data-driven tracker . For example , if the target only moves forward during training , the tracker may over fit to move straightly and fail to track other motion patterns , like a sharp turn . In this work , we propose a novel adversarial RL method for learning VAT , refereed to as AD-VAT ( Asymmetric Dueling mechanism for learning Visual Active Tracking ) . In the mechanism , the tracker and the target object , viewed as two learnable agents , are opponents and can mutually enhance during competition ( See Fig . 1 ) . As the training proceeds , the environments of AD-VAT naturally compose a curriculum , because the tracker is more likely to compete with a target with the appropriate difficulty level when both agents are becoming stronger simultaneously . When exploring the escape policy , the target consequently generates various trajectories to train the tracker . By the dueling/adversarial mechanism , the target is encouraged to discover the weakness of the tracker more often , which could serve as a kind of “ weakness-finding ” that makes the tracker more robust . However , in practice , using heuristic adversarial RL method for training VAT is unstable and slow to converge . To address these issues , we derive two components in AD-VAT : partial zero-sum reward ( PZR ) and tracker-aware model ( TAM ) for target . PZR is a hybrid reward structure . It encourages a zero-sum tracker-target competition in the near range , where the target is close to the expected position to tracker ; whereas , beyond the zero-sum zone , it is a non-zero-sum game , in which the target is penalized for running too far to track . Such reward structure is inspired by an observed phenomenon , that when the target quickly learns to be far away from the tracker while the tracker has no chance to see the target once more and henceforth gets plateaus during training . To learn the optimal policy to escape , we model the target with a “ tracker-aware network ” , i.e. , besides its own observation , the observation and actions of the tracker are also fed to the escaping network . In addition , to shape a better representation about the tracker , we add an auxiliary task for the target , learning to predict the immediate reward of the tracker . We argue that such an “ asymmetric dueling ” mechanism is able to learn a stronger target , which vice versa yields a more robust tracker ultimately . The experiment is conducted in various 2D and 3D environments for further studying AD-VAT . The 2D environment is a matrix map where obstacles are randomly placed . In the 2D environments , we evaluate and quantify the effectiveness of our approach in an ideal condition , free from noise in observation and action . We also conduct an ablation study to show the effectiveness of the two important components , “ partial zero-sum reward ” and “ tracker-aware network ” . The 3D environments are built on Unreal Engine 4 , a popular game engine for building high-fidelity environments . We choose a large room for training , where the texture of the background/players and the illumination are randomized . Three realistic scenarios built by artists are used for further evaluating the robustness . In the 3D environments , we further demonstrate that the tracker trained in AD-VAT is capable of generalizing to high-fidelity environments even it is trained in a simple environment . The contributions of our work can be summarized as follows : • We propose a novel Adversarial Reinforcement Learning method for VAT task , i.e. , the Asymmetric Dueling mechanism ( AD-VAT ) . In AD-VAT , the target learns to generate diverse trajectories when competing with the tracker , which in turn helps train a more robust tracker . • We provide two techniques to guarantee an efficient yet effective AD-VAT . 1 ) A partial zero-sum reward structure , which significantly stabilizes the training . 2 ) A tracker-aware network for the target , which yields better escaping policy and consequently better tracking policy . 2 RELATED WORK . Active Object Tracking . As described above that , active object tracking deals with object tracking and camera control at the same time . This problem attracts less attention compared with traditional object tracking ( or visual object tracking ) ( Wu et al. , 2013 ) . In general , this problem could be addressed in a two-step manner or in an end-to-end manner . In the two-step solution , traditional object tracking and camera control are conducted sequentially to obtain tracking results and manipulate camera . Great progress has been achieved in traditional object tracking in recent decades ( Babenko et al. , 2009 ; Ross et al. , 2008 ; Mei & Ling , 2009 ; Hu et al. , 2012 ; Bolme et al. , 2010 ; Kalal et al. , 2012 ) . Thus one can utilize mature visual tracking algorithms ( Henriques et al. , 2015 ; Ma et al. , 2015 ; Choi et al. , 2017 ) to accomplish the passive tracking task . According to the tracking results , camera control module could be developed to actively follow a target . For example , in ( Denzler & Paulus , 1994 ) a two-stage method is proposed to handle robot control by motion detection and motion tracking . Kim et al . ( Kim et al. , 2005 ) detect moving objects and track them using an active camera with pan/tilt/zoom . In ( Hong et al. , 2018 ) , two modules , a perception module and a control policy module , are learned separately to train an agent accomplishing both an obstacle avoidance task and a target following task . Admittedly , tracking algorithms have been successful while still not perfect . Camera control based on tracking results is challenging due to factors such as that the correspondence between image space and camera parameter space is unknown . Additionally , joint tuning of visual tracking and camera control is expensive and encounters trial-and-errors in real world . In end-to-end methods , direct mapping between raw input frame and camera action is established . Thus the intermediate visual tracking results are not necessarily required . For instance , by reinforcement learning , camera is controlled by signal outputted from a Conv-LSTM network given raw input frame ( Luo et al. , 2018 ; Luo et al. , 2019 ) . This end-to-end solution verifies the effectiveness , while not efficient enough to solve this problem . To improve the generalization potential , environment augmentation is conducted in this work . However , the target itself , like the path , motion pattern is fixed in their augmented environments , which is believed to limit the performance . This inspires us to resort to the idea of dueling in this paper , i.e. , the target learns to get rid of the tracker by itself , guiding the learning of the tracker . Adversarial Reinforcement Learning . Using an adversarial framework to improve the RL agent ’ s robustness is not a new concept . In ( Huang et al. , 2017 ) , they add nonrandom adversarial noise to state input for the purpose of altering or misdirecting policies . Mandlekar et al . ( Mandlekar et al. , 2017 ) use adversarial examples to actively choose perturbations during training in physical domains . In ( Pinto et al. , 2017 ) , an adversary is introduced to apply adversarial force during training . It produces an agent whose policy is robust to a wide range of noise at testing time . Sukhbaatar et al . ( Sukhbaatar et al. , 2017 ) pit a virtual agent Alice against another agent Bob , where Alice creates steadily more difficult challenges for Bob to complete . In ( Held et al. , 2017 ) , they optimize a generator network via adversarial training to make an agent automatically produce tasks that are always at the appropriate level of difficulty for the agent . Roughly speaking , in these proposed methods the adversary is viewed as a ghost/virtual player , which is unseen and could only challenge the protagonist by adding noise in the observation ( Huang et al. , 2017 ) , action ( Pinto et al. , 2017 ) and the system dynamics ( Mandlekar et al. , 2017 ) , or by generating the goal/initial position for navigation task ( Held et al. , 2017 ; Sukhbaatar et al. , 2017 ) . In this paper , we design a two-agent no-cooperative game for VAT task , where the tracker intends to lockup the target , while the target tries to escape from the tracker . Unlike the aforementioned previous work , the adversary ( target to be tracked ) in AD-VAT is a physical player , which could fully control the movement of the target at any time step . In this paper , such two-physical-player-competition is referred to as “ dueling ” . We argue that such a fully controllable opponent could bring to the protagonist more challenges during training and henceforth produces a more robust visual tracker . The proposed approach is also related to self-play ( Silver et al. , 2016 ; 2017 ; Bansal et al. , 2017 ) . In self-play , two physical players usually compete for the same goal , with symmetric observation and action space . Usually , both players share the same model so as to ensure each agent seeing an environmental dynamics at appropriate difficulty level . It is thus viewed as a means to automatically generate learning curriculum ( Bengio et al. , 2009 ; Kumar et al. , 2010 ) . Our setting is substantively different from standard self-play . In AD-VAT , the two players are asymmetric in observation and task . The target observes more , and is equipped with additional auxiliary task for its adversarial policy . Thus , the target and tracker could not share the same model while learning in AD-VAT , which would make the learning become unstable . To address the issue , we propose two techniques which stabilize the training , as described in Sec . 3 .

Summary: This paper presents a simple multi-agent Deep RL task where a moving tracker tries to follow a moving target. The tracker receives, from its own perspective, partially observed visual information o_t^{alpha} about the target (e.g., an image that may show the target) and the target receives both observations from its own perspective o_t^{beta} and a copy of the information from the tracker's perspective. Both agents are standard convnet + LSTM neural architectures trained using A3C and are evaluated in 2D and 3D environments. The reward function is not completely zero-sum, as the tracked agent's reward vanishes when it gets too far from a reference point in the maze.