Learning Distilled Collaboration Graph for Multi-Agent Perception

1 Introduction . Perception , which involves organizing , identifying and interpreting sensory information , is a crucial ability for intelligent agents to understand the environment . Single-agent perception [ 4 ] has been studied extensively in recent years , e.g. , 2D/3D object detection [ 18 , 27 ] , tracking [ 23 , 22 ] and segmentation [ 25 , 16 ] , etc . Despite its great progress , single-agent perception suffers from a number of shortcomings stemmed from its individual perspective . For example , in autonomous driving [ 8 ] , the LiDAR-based perception system can hardly perceive the target in the occluded or long-range areas . Intuitively , with an appropriate collaboration strategy , multi-agent perception could fundamentally upgrade the perception ability over single-agent perception . To design a collaboration strategy , current approaches mainly include raw-measurement-based early collaboration , output-based late collaboration and feature-based intermediate collaboration . ∗Corresponding authors . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Early collaboration [ 3 ] aggregates the raw measurements from all the agents , promoting a holistic perspective ; see Fig . 1 ( a ) . It can fundamentally solve the occlusion and long-range issues occurring in the single-agent perception ; however , it requires a lot of communication bandwidth . Contrarily , late collaboration aggregates each agent ’ s perception outputs . Although it is bandwidth-efficient , each individual perception output could be noisy and incomplete , causing unsatisfying fusion results . To deal with the performance-bandwidth trade-off , intermediate collaboration [ 19 , 34 , 20 ] has been proposed to aggregate intermediate features across agents ; see Fig . 1 ( b ) . Since we can squeeze representative information to compact features , this approach can potentially both achieve communication bandwidth efficiency and upgrade perception ability ; however , a bad design of collaboration strategy might cause information loss during feature abstraction and fusion , leading to limited improvement of the perception ability . To achieve an effective design of intermediate collaboration , we propose a distilled collaboration graph ( DiscoGraph ) to model the collaboration among agents . In DiscoGraph , each node is an agent with real-time pose information and each edge reflects the pair-wise collaboration between two agents . The proposed DiscoGraph is trainable , pose-aware , and adaptive to real-time measurements , reflecting dynamic collaboration among agents . It is novel from two aspects . First , from the training aspect , we propose a teacher-student framework to train DiscoGraph through knowledge distillation [ 1 , 10 , 26 ] ; see Fig . 1 ( c ) . Here the teacher model is based on early collaboration with holistic-view inputs and the student model is based on intermediate collaboration with single-view inputs . The knowledge-distillation-based framework enhances the training of DiscoGraph by constraining the postcollaboration feature maps in the student model to match the correspondences in the teacher model . With the guidance of both output-level supervision from perception and feature-level supervision from knowledge distillation , the distilled collaboration graph promotes better feature abstraction and aggregation , improving the performance-bandwidth trade-off . Second , from the modeling aspect , we propose a matrix-valued edge weight in DiscoGraph to reflect the collaboration strength with a high spatial resolution . In the matrix , each element represents the inter-agent attention at a specific spatial region . This design allows the agents to adaptively highlight the informative regions and strategically select appropriate partners to request supplementary information . During inference , we only need to use the student model . Since it leverages DiscoGraph as the key component , we call the student model as the distilled collaboration network ( DiscoNet ) . Multiple agents with the shared DiscoNet could collaboratively approach the performance of a hypothetical teacher model with the holistic view . To validate the proposed method , we build V2X-Sim 1.0 , a new large-scale multi-agent 3D object detection dataset in autonomous driving scenarios based on CARLA and SUMO co-simulation platform [ 6 ] . Comprehensive experiments conducted in 3D object detection [ 37 , 28 , 29 , 17 ] have shown that the proposed DiscoNet achieves better performance-bandwidth trade-off and lower communication latency than the state-of-the-art intermediate collaboration methods . 2 Multi-Agent Perception System with Distilled Collaboration Graph . This work considers a collaborative perception system to perceive a scene , where multiple agents can perceive and collaborate with each other through a broadcast communication channel . We assume each agent is provided with an accurate pose and the perceived measurements are well synchronized . Then , given a certain communication bandwidth , we aim to maximize the perception ability of each agent through optimizing a collaboration strategy . To design such a strategy , we consider a teacher-student training framework to integrate the strengths from both early and intermediate collaboration . During training , we leverage an early-collaboration model ( teacher ) to teach an intermediate-collaboration model ( student ) how to collaborate through knowledge distillation . Here the teacher model and the student model are shared by all the agents , but each agent would input raw measurements from its own view to either model ; see Fig . 2 . During inference , we only need the student model , where multiple agents with the shared student model could collaboratively approach the performance of the teacher model with the hypothetical holistic view . In this work , we focus on the perception task of LiDAR-based 3D object detection because the unifying 3D space naturally allows the aggregation of multiple LiDAR scans . Note that in principle the proposed method is generally-applicable in collaborative perception if there exists a unified space to aggregate raw data across multiple agents . 2.1 Student : Intermediate collaboration via graphs . Feature encoder . The functionality is to extract informative features from raw measurements for each agent . Let Xi be the 3D point cloud collected by the ith agent ( M agents in total ) , the feature of the ith agent is obtained as Fsi ← Θs ( Xi ) , where Θs ( · ) is the feature encoder shared by all the agents and the superscript s reflects the student mode . To implement the encoder , we convert a 3D point cloud to a bird ’ s-eye-view ( BEV ) map , which is amenable to classic 2D convolutions . Specifically , we quantize the 3D points into regular voxels and represent the 3D voxel lattice as a 2D pseudo-image , with the height dimension corresponding to image channels . Such a 2D image is virtually a BEV map , whose basic element is a cell that is associated with a binary vector along the vertical axis . Let Bi ∈ { 0 , 1 } K×K×C be the BEV map of the ith agent associated with Xi . With this map , we can apply four blocks composed of 2D convolutions , batch normalization and ReLU activation to gradually reduce the spatial dimension and increase the number of channels for the BEV , to obtain the feature map Fsi ∈ RK̄×K̄×C̄ with K̄ × K̄ the spatial resolution and C̄ the number of channels . Feature compression . To save the communication bandwidth , each agent could compress its feature map prior to transmission . Here we consider a 1 × 1 convolutional autoencoder [ 24 ] to compress/decompress the feature maps along the channel dimension . The autoencoder is trained together with the whole system , making the system work with limited collaboration information . Collaboration graph process . The functionality is to update the feature map through data transmission among the agents . The core component here is a collaboration graph1 GΠ ( V , EΠ ) , where V is the fixed node set and EΠ is the trainable edge set . Each node in V is an agent with the real-time pose information ; for instance , ξi ∈ se ( 3 ) is the ith agent ’ s pose in the global coordinate system ; and each edge in EΠ is trainable and models the collaboration between two agents , with Π an edge-weight encoder , reflecting the trainable collaboration strength between agents . LetMGΠ ( · ) be the collaboration process defined on the collaboration graph GΠ . The feature maps of all the agents after collaboration are { Hsi } Mi=1 ←MGΠ ( { Fsi } Mi=1 ) . This process has three stages : neural message transmission ( S1 ) , neural message attention ( S2 ) and neural message aggregation ( S3 ) . In the neural message transmission stage ( S1 ) , each agent transmits its BEV-based feature map to the other agents . Since the BEV-based feature map summarizes the information of each agent , we consider it as a neural message . In the neural message attention stage ( S2 ) , each agent receives others ’ neural messages and determines the matrix-valued edge weights , which reflect the importance of the neural message from one agent to another at each individual cell . Since each agent has its unique pose , we leverage the collaboration graph to achieve feature transformation across agents . For the ith agent , the transformed BEV-based feature map from the jth agent is then Fsj→i = Γj→i ( F s i ) ∈ RK̄×K̄×C̄ , where the transformation Γj→i ( · ) is based on two ego poses ξj and ξi . Now Fsj→i and Fsi are supported in the same coordinate system . To determine the edge weights , we use the edge encoder Π to correlate the ego feature map and the feature map from another agent ; that is , the matrix-valued edge weight from the jth agent to the ith agent is Wj→i = Π ( Fsj→i , F s i ) ∈ RK̄×K̄ , where Π concatenates two feature maps along the channel dimension and then uses four 1× 1 convolutional layers to gradually reduce the number of channels from 2C̄ to 1 . Meanwhile , there is a softmax operation applied at each cell in the feature map to normalize the edge weights across multiple agents . Note that previous works [ 19 , 20 , 34 ] generally consider a scalar-valued edge weight to reflect the overall collaboration strength between two agents ; while we consider a matrix-valued edge weight Wj→i , which models the collaboration strength from the jth agent to the ith agent with a K̄ × K̄ spatial resolution . In this matrix , each element corresponds to a cell in the BEV map , indicating a specific spatial region ; thus , this matrix reflects the spatial attention at a cell-level resolution . In the neural message aggregation stage ( S3 ) , each agent aggregates the feature maps from all the agents based on the normalized matrix-valued edge weights . The updated feature map of the agent i is Hsi = ∑M j=1 Wj→i Fsj→i , where denotes the dot product with channel-wise broadcasting . Remark . The proposed collaboration graph is trainable because each matrix-valued edge weight is a trainable matrix to reflect the agent-to-agent attention in a cell-level spatial resolution ; it is poseaware , empowering all the agents to work with the synchronized coordinate system ; furthermore , it is dynamic at each timestamp as each edge weight would adapt to the real-time neural messages . According to the proposed collaboration graph , the agents can discover the region requiring collaboration on the fly , and strategically select appropriate partners to request supplementary information . Decoder and header . After collaboration , each agent decodes the updated BEV-based feature map . The decoded feature map is Msi ← Ψs ( Hsi ) . To implement the decoder Ψs ( · ) , we progressively up-sample Hsi with four layers , where each layer first concatenates the previous feature map with the corresponding feature map in the encoder and then uses a 1× 1 convolutional operation to halve the number of channels . Finally , we use an output header to generate the final detection outputs , Ŷsi ← Φs ( Msi ) . To implement the header Ψs ( · ) , we use two branches of convolutional layers to classify the foreground-background categories and regress the bounding boxes .

Summary: This paper tackles the problem of 3D detection from LiDAR in the multi-agent setting. It has two main contributions. The first contribution is the new method, which uses knowledge distillation from a teacher network using early (sensor) fusion to individual student networks that use intermediate (feature) fusion. Student networks fuse features from other student networks using a learnable spatial attention while being trained using both a standard detection loss as well as a regularization to be close to the teacher networks’ intermediate features. The method outperforms baselines in both AP and communication bandwidth The second contribution is the release of the large scale synthetic dataset constructed with (SUMO and CARLA) used in the experiments.