## 1 Introduction  
Due to the rapid progress of 3D sensors (e.g., LIDAR scanners and RGB-D cameras), 3D point clouds[1, 2, 3] have become increasingly popular in domains such as AR/VR, autonomous driving, and robots. Unlike conventional 2D images, 3D point clouds provide richer depth information, making them successful for many applications. More information requires a higher computational load, leading to higher latency and hindering its application in real-time scenarios [4, 5, 6]. Therefore, efficiently processing these sparse data is a challenge, with tremendous potential value in the real world. The first challenge that arises when working with 3D point clouds is the overlapping regions between adjacent frames, as shown in Figure 1. Given that 3D point clouds are captured continuously using devices such as LiDAR, whose frequency is very fast, this makes consecutive frames have similar point distributions, particularly for stationary background instances due to their static characteristics. Consequently, processing consecutive frames involves significant redundant computation time on background and stationary instances, adversely affecting overall efficiency and inference time in real-world applications. Furthermore, each instance in point clouds comprises a varying number of points. The point count within different instances is inherently diverse, and background features occupy the majority of these points [7]. The imbalance in point distribution within instances poses challenges during both training and inference [8]. During training, models may become biased towards the majority classes (background instances) due to their overwhelming point count. This bias will bring suboptimal results in the detection and understanding of instances of interest, which often belong to minority classes. Similarly, computational resources are underutilized during inference due to significant time spent processing redundant information related to background and stationary instances. Addressing this challenge requires developing techniques to effectively manage point distribution imbalance and optimize point cloud processing. One potential approach is to utilize instance-aware sampling techniques during both training and inference to ensure a more balanced representation of instances. These techniques can prioritize instances of interest or dynamically adjust the sampling strategy based on the characteristics of the point cloud. Not only do point clouds' overlapping regions hamper training and inference, but their inherent characteristics also present substantial challenges for efficient computation. The hurdle when dealing with 3D point clouds lies in their unordered and sparse nature, unlike the structured format of 2D images. This fundamental difference challenges conventional convolutional neural networks (CNNs) adapted for grid-like data. Previous works have identified several approaches to address this issue. The first approach involves directly processing the point cloud data. The direct processing of point clouds requires the development of specialized network architectures capable of operating on unordered point sets, such as PointNet [9] and PointCNN [10]. While the approach mentioned above enables direct feature extraction on individual points, its computational efficiency is signifi  
Figure 1: Temporal correlation: significant overlaps exist in adjacent frames, and moving thing shows minor changes between frames. cantly hampered by the necessity of either handling each point separately or maintaining neighborhood information. Another technique converts the point cloud into a volumetric representation and divides the 3D space into a grid of voxels. Although this approach facilitates conventional 3D convolutional operations, it may be computationally intensive and impractical for real-time inference, particularly with large scenes [11]. The third widely recognized approach voxelizes the point cloud and subsequently applies sparse 3D convolution [12; 13; 14; 15].