Joint Homophily and Heterophily Relational Knowledge Distillation for Efficient and Compact 3D Object Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-31 10:53:01

Abstract: 3D Object Detection (3DOD) aims to accurately locate and identify 3D objects in point clouds, facing the challenge of balancing model performance with computational efficiency. Knowledge distillation emerges as a vital method for model compression in 3DOD, transferring knowledge from complex, larger models to smaller, efficient ones. However, the effectiveness of these methods is constrained by the intrinsic sparsity and structural complexity of point clouds. In this paper, we propose a novel methodology termed Joint Homophily and Heterophily Relational Knowledge Distillation (H2RKD) to distill robust relational knowledge in point clouds, thereby enhancing intra-object similarity and refining inter-object distinction. This unified strategy encompasses the integration of Collaborative Global Distillation (CGD) for distilling global relational knowledge across both distance and angular dimensions, and Separate Local Distillation (SLD) for a focused distillation of local relational dynamics. By seamlessly leveraging the relational dynamics within point clouds, the H2RKD facilitates a comprehensive knowledge transfer, significantly advancing 3D object detection capabilities. Extensive experiments on KITTI and unScenes datasets demonstrate the effectiveness of the proposed H2RKD.

Main paper: 
 introduction: 
 Lidar-based 3D Object Detection (3DOD) [20, 30] , leveraging point clouds for precise object identification and location, is a fundamental task in computer vision. Offering depth information beyond 2D detection, it's crucial for applications in robotics, autonomous vehicles, and augmented reality. However, increased detection capabilities come with higher computational costs. To mitigate this, Knowledge Distillation (KD) has been applied to balance performance with computational efficiency, transferring knowledge from larger to smaller models. Despite advancements in Knowledge Distillation (KD) [6, 28, 36] for streamlining image understanding models, their application to 3D Object Detection (3DOD) faces significant challenges. The primary obstacles stem from the inherent characteristics of point clouds, such as sparsity, irregularity, and geometric complexity, which hinder the generalization capabilities of these KD methods. To alleviate these issues, previous work [13] has focused on reducing the representational gap between teacher and student models by enhancing mutual information across pairs of regions, as illustrated in Figure 1 (a). To boost the transfer of structural information of objects, the other works tend to further distill relational knowledge. Concisely, PointDistiller [32] aims to distill the local geometric structures captured in K-Nearest Neighbor(KNN) graphs to uphold neighborhood relations, as shown in Figure 1(b) . However, the graph construction, heavily based on homophily, concentrates the learning within similar classes failing to embrace the multiple structural relationships present. More specifically, as Figure 2 demonstrates, the features extracted by the teacher show obvious similarity and difference relations, as marked by white and yellow boxes respectively. Unfortunately, after distillation, PointDistiller [30] only retains high similarity relations and significantly destroys most difference relations between voxel features. In contrast, our method emphasizes simultaneously distilling both relations to effectively imitate the teacher. Obviously, a robust relational knowledge should encompasses both homophily and heterophily, i.e., similarity relations within the same object and difference relations between different objects, respectively beneficial for enhancing the intra-class consistency and inter-class discrimination. Driven by the above analysis, in this paper, we propose a novel Joint Homophily and Heterophily Relational Knowledge Distillation method (H2RKD) for lidar-based 3D object detection, as illustrated in Figure 1(c ). H2RKD models and transfers relational knowledge in two ways, i.e., collaborative global distillation module (CGD) which distills global relations simultaneously, and separate local distillation module (SLD) which distills local homophily and heterophily separately. Specifically, CGD models distance-wise relations between pairs and angle-wise relations among triplets of features, implicitly collaborating both homophily and heterophily into global relations. Then two kinds of global relation consistency losses, including distance-wise relational knowledge distillation loss and angle-wise relational knowledge distillation loss, distill long-range semantic correlations buried in point clouds. Furthermore, to capture subtle dynamic local relations, SLD is proposed to separately embed local structure information into homophilic graphs and heterophilic graphs. Subsequently, SLD encodes and propagates intra-class and inter-class relations in dynamic graphs. Finally, a local relational knowledge distillation loss is adopted to distill local semantic relations and geometry information from teacher to student. Through the collaboration of CGD and SLD, our student model comprehensively learns the relational knowledge from the teacher, which preserves structural relations of point clouds, enhancing intra-class similarity and promoting inter-class discrimination simultaneously. We conduct extensive experiments on KITTI and large-scale nuScenes datasets to verify the effectiveness of the proposed approach.The main contributions can be summarized as follows:• We propose a novel Joint Homophily and Heterophily Relational Knowledge Distillation method (H2RKD) to distill robust relational knowledge in point clouds, thereby enhancing intra-object similarity and refining inter-object distinction. • We explore transferring homophily and heterophily relational knowledge in two ways: Collaborative Global Distillation module (CGD) distills global relational knowledge across both distance and angular dimensions, and Separate Local Distillation module (SLD) distills subtle local correlations and differences.• Extensive experiments demonstrate the effectiveness of the above contributions, and our proposed method achieves state-of-the-art performance on the challenging 3DOD task. 
related work 2.1 lidar-based 3d object detection: 
 Lidar-based 3D object detection aims to localize and classify 3D objects from point clouds. These methods can be briefly categorized as point-based [3, 21, 29] , pillar-based [12, 25] , and voxel-based [7, 38] .(1)Point-based methods [21] leveraged pointnet [17] or pointnet++ [18] to extract sparse point features for 3D object detectors. (2)As a Pillar-based method, [12] utilized the pointnet to learn the representations of multiple-pillar point clouds, and convert these representations into a pseudo image, which can be processed with 2D convolutional layers. (3)For the voxel-based method, [38] proposed a single-stage detector that divides point clouds into equally spaced 3D voxels and processes them with voxel feature encoding layers.Recently, some methods combining the aforementioned approaches have achieved notable performance, such as [20, 30] . However, as performance improves, lidar-based 3D detection models are likely to bury heavier computation costs. Hence, in this work, we focus on exploring knowledge distillation methods to boost the performance of lightweight 3D detectors.Knowledge distillation was originally proposed for model compression in [1] and focused on emulating the knowledge derived from a teacher network. Recently, knowledge distillation methods have demonstrated significant advancements in 2D object detection [23, 24, 33] and have also been leveraged to transfer knowledge in multi-modality setup [11, 37] or multi-frame to single-frame setup [35] in the 3D detection area.Existing knowledge distillation methods in lidar-based 3D detection tend to transfer knowledge representations or structural relationships within point clouds. In the first line, most methods prioritize transferring knowledge in crucial regions to acquire robust knowledge representations. [27] leverages cues in teacher prediction to determine the important areas for distillation. [13] maximizes the mutual information between intermediate features by bilaterally transferring. Another line is embedded with transferring structural relationships to learn more discriminative representations. [4] distill 3D representation under the consideration of the correlation among the multiple detection head components. [32] encodes the semantic information in the local geometric structure based on a local topology map using KNN.  
method 3.1 formulation and overview: 
 Problem Formulation. Given a set of point clouds 𝑋 = {𝑥 1 , 𝑥 2 , . . . , 𝑥 𝑛 } and the corresponding ground truth labels (GT labels) 𝑌 = {𝑦 1 , 𝑦 2 , . . . , 𝑦 𝑚 }, the 3D object detector can be formulated as 𝐷 = 𝐿 • 𝐻 , including feature encoding layers 𝐿 and detection head 𝐻 for prediction. In this paper, our goal is to train a student detector 𝑆 under the supervision of a pre-trained teacher detector 𝑇 , which is optimized by both 3DOD task loss and KD loss.Framework Overview. Figure 3 illustrates the architecture of the proposed H2RKD method, comprising three key components: teacher-student 3D object detection models, collaborative global distillation (CGD) module, and separate local distillation (SLD) module. During training, teacher and student models extract two sets of point cloud features, F𝑇 = {𝑡 𝑖 } 𝑀 𝑖=1 and F𝑆 = {𝑠 𝑖 } 𝑀 𝑖=1 , for total 𝑀 points/voxels. Based on F𝑆 , the student detection head predicts detection results, for calculating task loss and evaluation. To enhance feature robustness and simplify the relation complexity in F𝑇 and F𝑆 , we follow [32] to sample 𝑁 features 𝐹 𝑇 = {𝑡 𝑖 } 𝑁 𝑖=1 and 𝐹 𝑆 = {𝑠 𝑖 } 𝑁 𝑖=1 . Subsequently, relational knowledge is distilled in two ways: (1) CGD represents distance-wise and angle-wise relations within 𝐹 𝑇 or 𝐹 𝑆 by the Global Relation Modeling (GRM) module and distills global relations through distance-wise RKD loss and anglewise RKD loss, respectively. (2) SLD first constructs KNN graphs to model local relations within 𝐹 𝑇 and 𝐹 𝑆 , reconstructs them into homophilic graphs and heterophilic graphs by Separate Relation Modeling (SRM), and encodes and propagates relations within each graph. Then SLD aggregates homophily and heterophily within 𝐹 𝑇 or 𝐹 𝑆 and employs local RKD loss to distill these local relations. Finally, the overall framework is jointly optimized with 3DOD task loss, two global RKD losses, and a local RKD loss. During inference, the trained student model predicts detection results for performance evaluation.Subsequently, we provide a detailed description of the CGD module, SLD module and loss functions.Collaborative global distillation implicitly embeds homophily and heterophily into two global relations, including second-order and third-order relations within point clouds. Specifally, the secondorder relation is modeled by the distance-wise relation between pairs of features and the third-order relation is modeled by the angle-wise relation among ternary features.Drawing inspiration from the concept proposed in [15] , we employ two straightforward yet effective potential functions to capture these global relations, considering distance-wise and angle-wise relations. Correspondingly, we propose the distance-wise RKD loss and angle-wise RKD loss to distill the global relations.Global Relation Modeling. We define 𝜓 as a relational potential function for modeling each 𝜒 𝑁 , where 𝜒 𝑁 = {(𝑥 𝑖 , ..., 𝑥 𝑗 )|𝑖, 𝑗 ∈ 𝑁 , 𝑖 ≠ 𝑗 } represents a set of 𝑁 -tuples of distinct data. Thus, we (1) Given a pair of point/voxel features 𝜒 2 𝑇 from 𝐹 𝑇 , distance-wise potential function 𝜓 𝐷 measures the Euclidean distance between the two features in the representation space:EQUATION)where 𝜇 is a normalization factor for distance. To focus on relative distances among other pairs, we set 𝜇 to be the average distance between pairs from 𝜒 2 in the mini-batch.(2)Given a triplet of point/voxel features 𝜒 3 𝑇 = {(𝑡 𝑖 , 𝑡 𝑗 , 𝑡 𝑘 )|𝑖 ≠ 𝑗 ≠ 𝑘 } from 𝐹 𝑇 , an angle-wise relational potential measures the angle formed by the three examples in the representation space:EQUATIONwhere𝑒 𝑖 𝑗 = 𝑡 𝑖 -𝑡 𝑗 ∥𝑡 𝑖 -𝑡 𝑗 ∥ 2 , 𝑒 𝑘 𝑗 = 𝑡 𝑘 -𝑡 𝑗 ∥𝑡 𝑘 -𝑡 𝑗 ∥ 2 .Global Relation Distillation Loss. We leverage two kinds of global relation consistency losses: Distance-wise RKD loss and Angle-wise RKD loss to transfer global relations for improving the perception of similarity and distinction in point clouds.(1)Distance-wise relation is measured in both the teacher and the student, a distance-wise distillation loss is defined as:EQUATIONwhere (𝑡 𝑖 , 𝑡 𝑗 ) ∈ 𝜒 2 𝑇 and (𝑠 𝑖 , 𝑠 𝑗 ) ∈ 𝜒 2 𝑆 . (2)Angle-wise relation is measured in both the teacher and the student, an angle-wise distillation loss is defined as:EQUATIONwhere (𝑡 𝑖 , 𝑡 𝑗 , 𝑠 𝑗 ) ∈ 𝜒 3 𝑇 and (𝑠 𝑖 , 𝑠 𝑗 , 𝑠 𝑘 ) ∈ 𝜒 3 𝑆 . 𝑙 𝛿 is Huber loss [10] . The distance-wise distillation transfers the relationship of examples by penalizing distance differences between their representation, while the angle-wise distillation loss transfers the relationship of training example embeddings by penalizing angular differences. Therefore, the CGD implicitly distills both homophily and heterophily into global relations.Separate Local Distillation explicitly model local homophily and heterophily relations by homophilic graphs and heterophilic graphs. SLD encodes and propagates relations in each graph by a mixed filter and dynamic graph convolutional layers, respectively. Correspondingly, we propose the local knowledge distillation loss to distill the local relations.Separate Relation Modeling. Define graph data as 𝐺 = {V, Ã, 𝑓 }, where V represents the set of 𝑛 nodes, 𝑓 is the feature matrix from 𝐹 𝑇 or 𝐹 𝑆 . We initialize graph structure Ã based on these voxels or points clustered by KNN (K-Nearest Neighbours). The normalized adjacency matrix is𝐴 = 𝐷 -1 2 ( Ã + 𝐼 )𝐷 1 2, where 𝐷 represents the degree matrix. The corresponding graph Laplacian is 𝐿 = 𝐼 -𝐴.1.is a matrix with all 1. Subsequently, we will provide a detailed description of constructing two graphs.(1)Firstly, we construct a heterophilic graph by selecting the nodes that are far away from each other in both feature space and structure space as negative pairs. Specifically, we use complementary graphs of similarity graph 𝑊 and topology graph 𝐴 to construct a heterophilic graph H . The procedure is formulated as follows:EQUATION)where the similarity matrix 𝑊 is obtained through the cosine similarity of node features, which characterizes the closeness among nodes in feature space 𝑓 . ⊙ represents the Hadamard product, which is used to describe non-neighbor relations in both feature space and topology space.(2)Simultaneously, we could further improve the homophily level of the raw graph by minimizing the distances among adjacent nodes, which is formulated as: minEQUATIONwhere S 𝑖: represents the i-th row of S. Graph S will be more homophilic when edges are defined by nodes sharing high similarity. Furthermore, we use a regularization term to integrate the 1-hop and 2-hop neighbor relations. Let ∥𝑡 𝑖 -𝑡 𝑗 ∥ 2 = 𝐾 𝑖 𝑗 , then we construct a homophilic graph S by solving the following optimization problem: minEQUATION𝑖 𝑗 -S 𝑖 𝑗 ) 2 , 𝑠.𝑡 . S 𝑖 𝑗 > 0,EQUATIONwhere S (2) is the 2-hop graph, i.e.,S (2) = S × S.Extracting Dynamic Local Relation.We introduce a graph data mixture filter designed to handle diverse types of graphs, as follows:EQUATION)where 𝐿 S and 𝐿 H are the normalized Laplacian matrices of reconstructed homophilic and heterophilic graphs, and 𝑓 is the feature matrix. Then, we apply a dynamic graph convolution G as the aggregation operation upon the final representation F to align the dimensions of the F 𝑇 and F 𝑠 .Local Relation Distillation Loss. With the reweighting strategy, the local relation distillation can be formulated as:EQUATIONwhere 𝜙 𝑖 = 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (𝐼 ). 𝐼 represents the importance score acquired during the sampling of the N feature samples, as outlined in [32] . As shown in Figure 3 , our model is optimized by task loss and KD loss. The overall objective function is calculated as follows:EQUATIONThe task loss will be different for different models. For voxelsbased and point-based methods, task loss refers to [12, 30] and [21] , respectively. 
experiments 4.1 dataset and metrics: 
 Our experiments are conducted both on KITTI [8] and unScenes [2] , which consist of samples that have both lidar point clouds and images. Our models are trained with only the lidar point clouds. We have evaluated our method in both voxels-based object detectors PointPillars [12] and CenterPoint [30] , and the raw points-based object detector PointRCNN [21] . Following the PointPillars as the teacher network on KITTI, we use an AdamW optimizer [14] with a weight decay of 0.01 and a cyclic momentum update strategy to adjust the learning rate. We set 0.0001 for the initial learning rate, 1 for cyclic update time, and 0.90 for momentum. Following the Point-Pillars as the teacher network on unScenes, we use a step strategy to adjust the learning rate. The networks have been trained on RTX 3090 GPUs. All the experiments are conducted with mmdetection3d [5] and PyTorch [16] . We keep the evaluation settings in mmdetec-tion3d as default. The teacher model is the origin model before compression. The student model shares the same architecture and depth as its teacher but with fewer channels. We have mainly compared our methods with previous knowledge distillation methods, including methods proposed by Fitnets [19] , PATA [31] , CRD [22] , SE-SSD [36] , FBKD [34] , RDD [13] , PointDistiller [32] on KITTI, and CRD [22] , OFD [9] , MTS [26] , PointDistiller [32] on nuSecens.Results on KITTI. Experiments of 4× compressed PointPillars on KITTI. Table 1 shows the performance between our method and previous knowledge distillation method for BEV detection and 3D detection, respectively. Our proposed H2RKD method shows good performance, outperforming most existing methods. It is observed that on BEV and 3D detection, our method outperforms the second-best knowledge distillation method by 1.1% and 1.0% moderate mAP, respectively. Furthermore, our method empowers the student detector to surpass the teacher detector, resulting in performance improvements of 1.3% and 1.9% in BEV and 3D detection, respectively. Besides, our method attains the highest performance across all difficulty levels for car categories. It is worth mentioning that we performed optimally in each category on 3D detection. The above results validate the efficacy of our method in 3D Lidar-based object detection.Resluts on nuScenes. Experiments of around 2× and 4× compressed PointPillars and CenterPoint on nuScenes are shown in Table 4 . It is observed that our method leads to 0.5% and 0.6 % improvements on mAP and NDS on average, respectively for the compressed PointPillars model. Besides, the compressed Center-Point also shows 0.6 % improvements on NDS. These observations indicate that our method is also effective on the large-scale dataset.In summary, the effectiveness of our method can be demonstrated on both two datasets.Effects on different Detectors. Table 2 shows the performance of the voxel-based and point-based detectors trained with and without our method for BEV detection and 3D detection, respectively. On average, 2.6% and 1.5% moderate mAP improvements can be observed for the voxel and raw points-based detectors for BEV detection, respectively. Additionally, the voxel and raw points-based detectors exhibit moderate mAP improvements for 3D detection, averaging 2.6% and 0.7%, respectively. It demonstrates the advancements achieved by our approach in the voxel-based detector.Specifically, in BEV detection, our method demonstrated superior performance with the 4× compressed and accelerated PointPillars detector student surpassing its teacher by 1.3% mAP. Additionally, the 8× compressed and accelerated PointRCNN detector student, trained with our method, outperformed its teacher by 0.5% mAP. Similarly, for the 3D detection of PointPillars detectors, the 4× compressed and accelerated student, trained with our method, achieved a notable improvement of 1.9% mAP compared to its teacher. Furthermore, the compressed and accelerated PointRCNN detector student, trained with our method, exhibited enhancements of 0.7% and 0.6% in mAP on 3D detection, respectively.Consistent average precision boosts can be observed in the detection results of all categories. For instance, on BEV detection of 4× compressed PointPillars students, 0.3%, 4.4% and 3.7% mAP improvements can be observed for cars, pedestrians, and cyclists, respectively. Additionally, Consistent average precision boosts can be observed in the detection results of all difficulties. For instance, on 3D detection of 16× compressed PointRCNN students, 0.5%, 1.3%, and 1.1% mAP improvements can be observed for easy, moderate, and hard difficulties, respectively. In summary, these observations demonstrate that our method can successfully transfer teacher knowledge to the voxel-based and point-based student detectors. Furthermore, it also validates that our method is capable of learning an effective and lightweight 3D detector.Effects of CGD and SLD Modules. Our H2RKD is mainly composed of two components, including collaborative global distillation (CGD) and separate local distillation (SLD). Ablation studies with 4× compressed PointPillars students on KITTI are shown in Table 3 . We also compare with the local distillation in [12] .It is observed that on BEV detection and 3D detection, 2.3% and 2.2% mAP improvements can be obtained by only using CGD to distill the global relation within the point cloud, respectively. Moreover, the category of cyclists on easy demonstrates its optimal performance solely with the application of the CGD module.Additionally, 2.2% and 2.3% mAP boosts can be gained by using SLD on BEV detection and 3D detection, respectively. Moreover, the pedestrian category on easy and moderate demonstrate the optimal performance solely using the SLD module, respectively. Furthermore, Compared to the approach of solely constructing the KNN graph, the SLD module enhances its performance by 0.4% and 1.3% in BEV and 3D detection, respectively. From the aforementioned analysis, we observe that the SLD module plays a more significant role in 3D detection, indicating the importance of local structural relationships among point clouds.In summary, these observations indicate that each module in H2RKD has its individual effectiveness and their merits are orthogonal. Besides, the CGD module proves to be more effective in BEV detection, whereas the SLD module demonstrates greater efficacy in 3D detection.In this subsection, we have visualized the detection results of the student model trained with and without our method, as shown in Figure 4 . Furthermore, we visualized the detection results and conducted a comparison with the ground truth, as shown in Figure 5 . Note that both student models are 4× compressed PointPillars trained on KITTI. The green boxes indicate the boxes of the model prediction. The visualization clearly demonstrates the strengths of our approach. Specifically, as shown in Figure 4 , a model with our H2RKD exhibits the capability to detect object in distant regions thanks to our global relation distillation. when compared with the ground truth shown in Figure 5 , our method proficiently identified the majority of objects and gained substantial local knowledge, as indicated by the red points. 
conclusion: 
 In this paper, we aim to simultaneously transfer both homophily and heterophily relational knowledge of point clouds, enhancing intraobject similarity and inter-object discrimination. To this end, we propose a novel joint homophily and heterophily relational knowledge distillation method(H2RKD), which distills the relational knowledge by collaborative global distillation (CGD) and separate local distillation (SLD). Specifically, CGD transfers both distance-wise and angle-wise global relations, implicitly collaborating homophily and heterophily. To further transfer subtle correlations and differences, SLD explicitly distills local homophily and heterophily by reconstructed graphs, separately. Extensive experiments on KITTI and unScenes datasets demonstrate the effectiveness of the proposed H2RKD.  
 
 Appendix:  


Decision: The paper has been thoroughly reviewed by four reviewers, all of whom agree on its acceptance. The strengths of this work lie in its novel insights into multi-dimensional relational distillation from both global and local perspectives, as well as homogeneity and heterogeneity. Furthermore, the method has been extensively validated, and its high performance demonstrates the effectiveness of the approach.

However, the authors must address several points, primarily related to writing and illustrations, before submitting the final version. Despite these shortcomings, the overall merits of the paper justify its acceptance, as recommended by the ACs. To ensure the quality of the work, the authors are advised to make significant improvements addressing all reviewers' concerns.

Reviews: review: paper_summary: paper_summary: This paper focus on Knowledge Distillation for Lidar-based 3D Object Detection. The authors obverse that existing work is ineffective at distilling relationships with low similarity between features from the teacher to the student. Based on this finding, they proposed Collaborative Global Distillation and Separate Local Distillation to more comprehensively transfer the teacher's knowledge to the student. The experimental results on various datasets demonstrate the effectiveness of their methods.
strength_weakness: strength_weakness: In fact, I am not familiar with this field. Therefore, please take my opinion into consideration cautiously.

1. The writing is clear and easy to understand, the experiments are relatively thorough, and the results are quite satisfactory.
2. The motivation is reasonable, and the proposed methods are relevant to the motivation.
3.  The proposed methods seem to have a certain degree of novelty and are relatively comprehensive. For example, the authors approach relational distillation from both global and local perspectives. For the global aspect, they consider distances and angles between features for distillation, while for the local aspect, they take into account Homophily and Heterophily Relational Knowledge.
limitations: limitations: Since I am not familiar with this field, I cannot summarize any meaningful limitations. Please refer to the opinions of other reviewers.
paper_summary: paper_summary: This paper proposes a novel methodology termed Joint Homophily and Heterophily Relational Knowledge Distillation (H2RKD) to distill robust relational knowledge in point clouds, thereby enhancing intra-object similarity and refining inter-object distinction. H2RKD consists of Collaborative Global Distillation (CGD) and Separate Local Distillation (SLD). The former is designed for distilling global relational knowledge across both distance and angular dimensions, and the latter is responsible for a focused distillation of local relational dynamics.
strength_weakness: strength_weakness: 1. The authors focus on multi-dimensional relational distillation for 3D object detection, such as global and local, homophily and heterophily.
2. Experimental comparison is sufficient. The proposed method achieves SOTA performance.
limitations: limitations: 1. There is confusion about inter-object and intra-object are used in the abstract, and inter-class and intra-class are used in the introduction.
2. A suggestion is that a subsection is needed to be added to organize the symbols.
3. Minor issue: L = I (i.) - A  in line 407.
paper_summary: paper_summary: This paper propose a novel Joint Homophily and Heterophily Relational Knowledge Distillation method for 3D Object Detection, which contains the Collaborative Global Distillation and Separate Local Distillation module to distill global relational knowledge and s subtle local correlations, respectively. Extensive experiments demonstrate the effectiveness of the proposed methods.
strength_weakness: strength_weakness: - The proposed method achieve SOTA performance against previous knowledge distillation methods on 3D detection.
- The motivation is clear, allowing the student to simultaneously learn from teacher about similar relationships (homogeneity) of the same object and different relationships (heterogeneity) of different objects.
- The experiments are solid.
- The paper is well-written and easy to follow.
limitations: limitations: - In the Collaborative Global Distillation, the proposed method uses distance-wise RKD loss and angle-wise RKD loss to distill the global relationship between students and teachers, which is very similar to the RKD method, and the authors need to give a detailed explanation about the difference with the previous method. 
- The main motivation of the proposed method is to consider homogeneity and heterogeneity in relationship distillation. However, the motivation of the proposed Collaborative Global Distillation mainly focus on the global information transfer, and does not appear to consider homogeneity and heterogeneity, which may deviate from the original motivation. This point may weaken the contribution of this work.
paper_summary: paper_summary: This paper proposes a new method called Joint Homophilic and Heterophilic Relational Knowledge Distillation (H2RKD) for distilling robust relational knowledge in point clouds to enhance similarities within objects and refine distinctions between objects. This unified strategy consists of integrating Collaborative Global Distillation (CGD), which is used to distill global relational knowledge across distance and angle dimensions, and Separate Local Distillation (SLD), which is used to centrally distill local relational dynamics. By seamlessly utilizing relational dynamics in point clouds, H2RKD facilitates comprehensive knowledge transfer and greatly improves 3D object detection.
strength_weakness: strength_weakness: The novelty of this paper is outstanding, making enough innovations to the technique of knowledge distillation in point clouds, by dividing and synthesizing the Homophily and Heterophily.

The paper is theoretically and experimentally detailed, through theoretical derivation of collaborative global distillation and separate local distillation, and embedded in a network of teachers and students.

The quality of the paper's presentation is excellent overall, but some of the expressions could be improved. In addition, there are several obvious factual errors in the paper.

The method proposed in this paper is an incremental component that needs to be embedded on existing methods to show specific gains, which may be a hindrance for real-world applications.
limitations: limitations: 1. The font size in the illustrative figures in the paper needs to be adjusted for better presentation quality, especially in Figures 1 and 2.

2. The contribution mentions intra-object similarity and inter-object discrimination, which are widely used in self-supervised learning and transfer learning, however I did not find any substantial innovations made by the authors on them in the paper.

3. The framework in Figure 3 is in my opinion brilliant, however the captions could be improved. Suggest "The framework of the proposed joint homophily and heterophily relational knowledge distillation method (H2RKD) for 3D object detection." can be changed to "The framework of the proposed H2RKD for 3D object detection.". In addition, the icon in front of "Angle Relation" needs to be further adjusted.

4. There are irregularities in the case of symbols in Eqs. 7-9 in the paper, which need to be carefully reviewed by the authors.

5. Why are some of the methods in the results table set in italics?

6. There are some obvious factual or grammatical errors in the paper, such as:
"unScenes", "K-Nearest Neighbor (KNN)", "(1) Given or (2) Given", "Car or Pedestrians", "operations(/G) .", "y.L", etc. Note that some of the errors occur multiple times, the authors are asked to search globally in order to handle them appropriately.

7. Knowledge distillation and transfer learning mentioned in the paper are popular techniques, and the authors are suggested to supplement the methods and experiments in the related literature: "Self-supervised intra-modal and cross-modal contrastive learning for point cloud understanding" , TMM 2023; "M3SOT: Multi-Frame, Multi-Field, Multi-Space 3D Single Object Tracking", AAAI 2024; "Joint Semantic Segmentation using representations of LiDAR point clouds and camera images", INFFUS 2024.