LabelFormer: Object Trajectory Reﬁnement for Offboard Perception from LiDAR Point Clouds
Authors: Anqi Joyce Yang
Sergio Casas
Nikita Dvornik
Sean Segal
Yuwen Xiong
Jordan Sir Kwang Hu
Carter Fang
Raquel Urtasun
Venue: CoRL 2023 Poster (2023)
Keywords: Auto-labelling
Offboard Perception
Trajectory Refinement
Transformer

Abstract: A major bottleneck to scaling-up training of self-driving perception systems are the human annotations required for supervision. A promising alternative is to leverage “auto-labelling” offboard perception models that are trained to automatically generate annotations from raw LiDAR point clouds at a fraction of the cost. Auto-labels are most commonly generated via a two-stage approach – first objects are detected and tracked over time, and then each object trajectory is passed to a learned refinement model to improve accuracy. Since existing refinement models are overly complex and lack advanced temporal reasoning capabilities, in this work we propose LabelFormer, a simple, efficient, and effective trajectory-level refinement approach. Our approach first encodes each frame’s observations separately, then exploits self-attention to reason about the trajectory with full temporal context, and finally decodes the reﬁned object size and per-frame poses. Evaluation on both urban and highway datasets demonstrates that LabelFormer outperforms existing works by a large margin. Finally, we show that training on a dataset augmented with auto-labels generated by our method leads to improved downstream detection performance compared to existing methods. Please visit the project website for details https://waabi.ai/labelformer/.

LabelFormer: Object Trajectory Refinement for Offboard Perception from LiDAR Point Clouds
Anqi Joyce Yang 1.2
Sergio Casas 1.2
Nikita Dvornik'
Sean Segal 1.2 Raquel Urtasun
Yuwen Xiong"
Jordan Sir Kwang Hu'
Carter Fang'
Waabi University of Toronto²
(jyang, sergio, ndvornik, ssegal, jkhu, cfang, Ortasun)[aaalle ai
Abstract: A major bottleneck to scaling-up training of self-driving perception systems are the human annotations required for supervision. A promising alter- native is to leverage "auto-labelling" offboard perception models that are trained to automatically generate annotations from raw LiDAR point clouds at a fraction of the cost. Auto-labels are most commonly generated via a two-stage approach
first objects are detected and tracked over time, and then each object trajec- tory is passed to learned refinement model improve accuracy. Since existing refinement models are overly complex and lack advanced temporal reasoning ca- pabilities, in this work we propose LabelFormer, simple, efficient, and effective trajectory-level refinement approach. Our approach first encodes each frame's s observations separately. then exploits self-attention to reason about the trajectory with full temporal context, and finally decodes the refined object size and per- frame poses. Evaluation on both urban and highway datasets demonstrates that LabelFormer outperforms existing works by large margin. Finally, we show that training a dataset augmented with auto-labels generated by method leads to improved downstream detection performance compared to existing methods. Please visit the project website for details sttps://waalds ai ai/labelformer/
Keywords: Auto-label, Offboard Perception, Trajectory Refinement, Transformer
Introduction
Modern self-driving systems often rely large-scale manually annotated datasets to train object detectors to perceive the traffic participants in the scene. Recently, there has been a growing interest in auto luto-labelling approaches that can automatically generate labels from sensor data. If the comput- ing cost of auto-labelling is lower than the cost of human annotation and the produced labels are of similar quality, then auto-labelling can be used to generate much larger datasets at a fraction of the cost. These auto-labelled datasets can in turn be used to train more accurate perception models.
Following [1, 2]. we use LiDAR as input as it is the primary sensor deployed on many self-driving platforms [3, 4]. In addition, we focus on the supervised setting where set of ground-truth labels are available to train the auto-labeller. This problem setting is also referred to as offboard perception [2], which, unlike onboard perception, has access to future observations and does not have real-time constraints. Inspired by the human annotation workflow, the most common paradigm [1, 2] tackles the offboard perception problem in two stages, as shown in Fig. 1. First, objects and their coarse bounding box trajectories are obtained using "detect-then-track framework, and then each object track is refined independently. The main goal of the first stage is to track as many objects in the scene as possible (i.e. to achieve high recall), while the second stage focuses track refinement to produce bounding boxes of higher quality. In this paper, we focus on the second stage, which we refer to as trajectory refinement. This task challenging as it requires handling object occlusions, the sparsity of observations as range increases, and the diverse size and motion profiles objects. In order to handle these challenges, it is key to design model that is able effectively and effi- ciently exploit the temporal context of the entire object trajectory. However, existing methods [1, 2]
Work done at Waabi.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

First Stage: Coarse Initialization
Second Stage: Trajectory Refinement
Object
Detect
Track
Tracks
Frame
Object N
Point cloud sequunica
Rounding boves track
Fnttal object besed
Refredd ob act baced tooks
(colons represent track Ds
point obsemvaiios in global frame
Figure I:
Two-stage auto-labelling paradigm. The first stage uses a detect-then- thackn paradigm to obtain coarse object trajectories. The second stage refines each trajectory independently.
fall short as they are designed to process trajectories from dynamic objects in a sub-optimal sliding window fashion, where neural network is applied independently each time step over limited temporal context to extract features. This is inefficient as features from the same frame are extracted multiple times for different overlapping windows. As a consequence, the architectures exploit very small temporal context to remain within the computational budget. Furthermore, prior works uti- lized complicated pipelines with multiple separate networks (e.g., to handle static and dynamic objects differently), which are hard to implement, debug, and maintain.
In this paper, we take a different approach and propose LabelFormer, simple, efficient, and ef- fective trajectory refinement method. It leverages the full temporal context and results in more accurate bounding boxes. Moreover, our approach is more computationally efficient than the ex- isting window-based methods, giving auto-labelling a clear advantage over human annotation. To achieve this goal, we design a transformer-basee architecture [5], where we first encode the initial bounding box parameters and the LiDAR observations each time step independently, and then utilize self-attention blocks exploit dependencies across time. Since method refines the entire trajectory in single-shot fashion, it only needs to be applied once for each object track during in- ference without redundant computation. Additionally, our architecture naturally handles both static and dynamic objects, and it is much simpler than other approaches.
Our thorough experimental evaluation both urban and highway datasets show that approach not only achieves better performance than window-batt based methods but also is much faster. Moreover, we show that LabelFormer can be used to auto-label a larger dataset for training downstream object detectors, which results in more accurate detections comparing to training on human data alone or leveraging other auto-labellers.
2 Related Works
biseR- Auto-Labelling: These approaches have emerged from the need automate the
expensive human labelling process. Multiple approaches [6. 7, 8, 9, 10, 11, 12, 13] have attempted to generate auto-labels with little no supervision, but they do not achieve satisfactory performance at high precision. More related to our method, pioneering works Auto4D [1] and 3DAL [2] devel- oped a two-stage paradigm which first uses an object detector followed by a multi-object tracker to generate coarse object trajectories, and then refines each object trajectory separately using a super- vised model. For the second-stage trajectory refinement, which is the problem we focus on this work, Auto4D [1] consists of two separate models. It first trains size branch that refines only the size with full temporal context, and then freezes the refined size and trains motion path network to refine each frame's pose with a small local window. 3DAL [2] first trains motion classifier, and then employs two separate networks for stationary and dynamic objects. The stationary network uses observations from the full trajectory, while the dynamic network again operates in a windowed fashion that consumes point cloud observations from a very small local window. As a result, both works have limited temporal context, incur heavier computational costs from overlapping windows, and involve complicated workflows with at least two models that cannot be trained jointly. In con- trast, our work designs single network to jointly optimize the entire bounding box trajectory at once. Finally, concurrent works [14, 15] also ingest the full object trajectory for refinement, but [14]

---

aggregates all point observations in the global frame and does not conduct explicit cross-frame frame tem- poral reasoning, and [15] trains three separate models while we only need one.
Sequence processing with Transformers: The transformer architecture [5] is the state-of-the-art model for sequence processing. It incorporates a self-attention mechanism that models communica- tion between the sequence elements and enables sequence-level reasoning. Transformers have been successfully applied in various domains, including language modeling [5, 16, 17], image and video processing [18, 19), object tracking [20), and 3D understanding [21, 22). They are also effective for multi-modal problems like vision- laiiuale joint modeling [23] and autonomous driving [24]. The application of transformers particularly advantageous for modeling sequences with long-range dependencies, which preciscly the case for our trajectory-Ieve refinement task. To handle long sequences, it is common to preprocess the input tokens with powerful short-range encoder and then utilize a transformer to perform global reasoning on the encoder's outputs [25, 26, 27]. For instance, the work of [25] addresses long video modeling by processing short video snippets with object detection, tracking, and action recognition models before passing their outputs to a trans- former for holistic video analysis. Drawing inspiration from these works, our LabelFormer takes the per-frame bounding box initializations obtained with a detector and tracker, and refines them by jointly considering the bounding box trajectories and point clouds of the entire trajectory sequence.
3 LabelFormer: Transformer-basee Trajectory-level Refinement
The goal of trajectory refinement is to produce an accurate bounding box trajectory given a noisy initialization that is typically obtained using a detect-then-track paradigm. In this paper we propose a novel transformer-based architecture that takes the raw sensor observations and the full object trajectory as input, and conducts temporal reasoning simultaneously for all frames in the trajectory. This architecture naturally handles both static and dynamic objects and jointly refines the bounding box size and the motion path, resulting in much simpler, efficient and effective approach.
3.1 Problem Setting
The input to a LiDARRbbaed auto-labeller a sequence of T "frames" of point clouds, ob- tained from single LiDAR scan (i.e., 360 sweep). Since Bird's s-ye View (BEV) is the de- facto representation for downstream tasks in self-driving, such motion and occupancy forecast- ing [28, 29, 30, 31, 32] and motion planning [33, 34, 35, 36), LabelFormer operates in BEV.
In the first stage of the auto-labelling pipeline, we use a detection model followed by a multi- object tracker, which provide us with N perceived objects with initial bounding box trajecto- ries (B" IB.....,DI ). Each object trajectory B (we omit the object index for brevity) is defined by sequence of M bounding boxes B (b....., bM), where each BEV bounding box b, (I:, ye, 4, wi, 0,) is parameterized by center position (I,, Hc), bounding box length and width (4,, w.:) and heading angle 0,. All bounding box poses (I; Vs, 6,) are in a trajectory coordinate frame that is centered at the middle of the trajectory, i.e., (Im, Vm. 0m) (0, 0, 0) with m = M//2 as the middle index. Note that the sequence length M may vary across objects, and that the bounding box dimensions (4, wx) for the same object might be different across frames because object detec- tors output bounding boxes for each frame separately. addition, each bounding box b, is detected from scene-level point cloud P. E R that consists of nı points, and each point is represented as its 3D position in the same trajectory frame along with its timestamp.
Given each object's coarse bounding box trajectory B = (b,.... b,,) and the scene-level point clouds (P,....,,P) PM). the goal of trajectory refinement is then to output a precise trajectory B (b.....,b .b)) with bounding box size (i, ②) that is shared across the entire trajectory.
3.2 Model Architecture
To refine the entire actor trajectory with full temporal context, LabelFormer first uses a shared encoder to process each frame's 's observations separately, and then leverages self-attention to reason across frames. Finally, decoder is employed to obtain the refined pose at each frame well as consistent bounding box size for the full duration of the trajectory. Fig. 2 illustrates the architecture.
3

---

Croes Attention
Motion Decoder
P.Y.
point pold
cloud
teature
Frame
M.y.
y.
Reee*
point cloud
welrre
Mi P
Frame M
K.y.
box festure
A/x y.
CNN
point eloud
point reature
MeanFocl + MLP
LabelFormer Architecture which first encodes box and point observations for each frame separately, then applies stack of self-attention layers among per-frame features, and finally decode a size residual along with per-frame pose residuals.
Figure 2:
3.2.1 Per-Frame Encoder
Given the initial object BEV bounding box trajectory B (bI...., bas) and point clouds (PI...., PMr) the trajectory frame described in the problem setting, we first extract object points inside each bounding box b, by filtering the respective scene-level point cloud P, in BEV. That is, we only keep the points in P: whose BEV projection lies inside the 2D BEV bounding box b,. Since the bounding box initialization b; is noisy, similar to previous works [1, 2]. we enlarge the filtering region by 10% such that the point cloud contains the full object with high probability. As result, each frame has two input observations: the initial BEV bounding box b, E RR and the set of m; 3D object points O, E R x4, We next encode each frame's bounding box and point observations separately before fusing them. For brevity, we refer "bounding box" "box" from now on.
Bounding Box Encoding: Since the detector might produce noisy heading directions, we first pre- process the heading directions with a simple heuristic that flips inconsistent headings by 180 based on majority voting. We next use simple multi-layer perceptron (MLP) that maps box parameters b; with updated heading directions high-dimensional features a; MLP(bb.) R.D.
Previous works [1, 2] aggregate multi-frame points the global frame for feature extraction. However, in practice, humans estimate the relative transformation between two point clouds by aligning them in the object frame. Motivated by this fact, we use the object pose initialization b; transform O, from the trajectory frame to the object frame. We next learn representation of the object-frame points with PointPillars [37]-style encoder. Specifically, first voxelize the object-frame points into a N₂ x N, x N; grid, apply PointNet [38] to all points in each 3D voxel grid to extract per-voxel feature, and fuse all features along height to generate BEV feature map E IR N₂ x N., D., We next feed the BEV voxel feature map into a 2D convolutional network (CNN) composed of multi-scale ResNet [39] backbone followed by FPN [40] to obtain 4x downsampled feature map F; E N Since the receptive field of the CNN is designed to cover the entire object space, to retrieve point feature p; E R we simply index the feature map F; at its spatial center.
Point Cloud Encoding:
Finally, we fuse the box feature a, and point feature p, to derive the final frame- wise feature f; = a; + MLP(p:)) E TR.. We apply the same encoder with shared weights on each frame individually and obtain a set of frame-wise feature tokens (f.).ss...
Feature Fusion:
3.2.2 Trajectory-leve Understanding via Cross-Frame Attention
Intuitively, both box parameters and point observations useful for trajectory refinement. How- ever, traditional path smoothers and point cloud registration methods based ICP [41, 42] fail to fuse the information from both sources. In addition, ICP reasons on the point level and fails

---

when the points are sparse or the point cloud pair has small overlaps. Structured optimization- approaches [7, 8] that combine both methods operate in an online fashion with limited temporal con- text, and still suffer from ICP's failure modes. To address these limitations from traditional methods, we exploit the fused per-frame features (fi,..., ., fMr) ) and model relationships between frames at the feature level via self-attention [5]. The attention module allows for efficient pairwise reasoning across frames and offers the flexibility to operate on sequences with arbitrary length.
At a high level, the input to the attention module is the feature sequence (fi...., fax), and the out- put (g) ..... ) represents the updated per-frame features after absorbing information from the entire sequence. In particular, the attention module contains L attention blocks, where the Ith attention block consumes the previous block's output feature sequence (g) (g(--1)... ,.....M. B.M B
and generates updated feature sequence (gl ..... 8.M B with the first attention block input (0) f.. Each attention block contains a self-attention layer followed by feed-forward MLP. For each input feature vector g g/²²1) RD, the pre-norm self-attention mechanism first applies LayerNorm [43] (LN) followed by three separate linear projections derive query, key and value vectors q, k.") v(0) E IP respectively. We stack the keys and values from all M frames to form matrices KO), VG) E RRM Then, for each query frame i, we compute the attention scores between frame i and all frames by comparing query vector (i) and each key in K();
g;
K(
(3) a
softmax
+ W;
ERRM
Vd
where d4 is a scaling factor. Note that in Eq. I we adopt AliBi [44], which is form relative po- sitional encoding that leverages positional difference between query and key frames. AliBi directly adds weighted biases w, IR- (with way -m l jl. m is a fixed constant) to the dot product attention score map and is shown to generalize better to longer sequences at test time.
With the attention scores, we can then obtain aggregated feature with h² a. vt) E ER
We then apply the subsequent MLP layer to derive output features g, of the j attention block:
h (t) LN G h.²)
g MIP((IN( (()) e R..
In practice, we use multi-head self-attention to increase expressivity, which partitions the D- dimensional input feature vector into H groups, employs separate self-attention head for each feature group, and concatenates the output features from each attention head as the final feature. Please refer to [5] for more details multi attention. After L chained self-attention blocks, the attention module outputs a sequence of updated feature vectors at each frame (g("), ......M g()) which is used to decode the final bounding box trajectory.
(g("),......
3.2.3 Motion Path and Size Decoder
Given the feature sequence (gi ..... gir ............ ), we decode the final motion path trajectory and object size which is consistent for the entire trajectory. To decode the refined bounding box pose at each frame, we simply feed the feature ( into an MLP to obtain pose residual (Az;, Ay, A0,) and sum them with the initialization b, to obtain the final refined pose parameters (æz, De. 0.) (z; + Ar; y: + Ay; 02 + A.:). To decode the refined object size, leverage context from all frames via mean pooling, and use an MIP to obtain a size residual (Al, Aw) MLP(meenn(F) (gg (7.) })). We compute the final refined object size as (I, w) = (mean(14)) + Al, mean(((.)) + Aw). The final refined bounding box at each frame will be b, = (0., 0i, i, w. 0.).
(L)),
3.3 Training
We train the entire model (i.e,, encoder, attention module, decoder) end-to-end by minimizing combination of regression loss that directly compares the refined box parameters b, with ground- truth box parameters bj, and an loU-based loss that compares the axis-aligned bounding boxes:
T(((..). (b;)) (ree((()) (b/3)) Loo ((b.), (b;)).
(3)
5

---

Please see supp. for more details. We apply two forms of data augmentation during training: (I) we randomly sample a subsequence of the input actor trajectory, and (2) we independently perturb each initial bounding box by applying a translational offset uniformly sampled from [-0.25, 0.25 m for a and y each, a rotational offset uniformly sampled from 10, 10] degrees, and an offset uniformly sampled from |max(-0.22 - )), min(0.2, 2)] and [max( 1m.11- w: sions. Note that the offsets sampled per frame and applied to each bounding box separately.
2 m0.1.. 2-) )] for the dimen-
4 Experiments
In this section, we evaluate the effectiveness of our approach on two real-world datasets. First, we describe the experimental setting and metrics used for evaluation. Next, we show that method outperforms previous works on the trajectory refinement task for multiple initializations. Further- more, we demonstrate that the improved refinement translates to downstream detection performance when training with auto-labels, showcasing that our approach can be used to train better object detectors. Finally, we conduct thorough ablation studies to analyze the effect of our design choices.
Datasets: We use two datasets to evaluate our method in both urhan and highway domains, which include object trajectories with diverse motion profiles. For the urban setting, we use the Argoverse 2 Sensor dataser [3] (AV2) that was collected in six distinet US cities. AV2 contains 850 15-second long snippets and around 65.7k vehicle trajectories. The LiDAR data is fairly sparse as it is captured by two 32-beam LiDARs that spin at 10Hz in the same direction but are 180 apart in orientation, and we aggregate both LiDAR scans in the same sweep interval to form an input frame. We use the official train and validation splits with 700 and 150 snippets each. For the highway setting, we use an in-house Highway dataset, which contains 188 20-second long snippets collected from US highways with roughly 5.8k vehicle trajectories. The LiDAR data in this dataset is denser as it comes from 128-beam LiDAR sensor that spins at 10Hz. We split the dataset into 150 training snippets and 38 validation snippets. For our experiments in both datasets, we focus auto-labelling vehicles in the scene, with a detection region of interest of [-125, 200] meters longitudinally and [-50, 50) meters laterally with respect to the ego vehicle's traveling direction.
Metries:
Following [1], for each object k we compute the track-level IoU S* 1 En loU ; )²), where M& is the trajectory length and B B E IR'S are the respec- M.
tive ground-truth truth and refined BEV bounding box at each frame i. To aggregate across all N ob- ject trajectories, we report the mean IoU as E Sh. To understand coverage, we addition- ally report average recall at various IoU thresholds, i.e., RC@a = E 1(5 2 a), where [1 i 2
is the indicator function. In our results we choose IoU thresholds 1(x 2 a) = 0 otherwise
0.5, 0.6, 0.7.
Implementation details: For the box encoder, we use single linear layer to map the 5 box parameters to an output dimension, D = 256. For the attention module, we use L = 6 attention blocks, with H = attention heads. Both the size and pose decoders are a single linear layer. For both datasets, we train our model with the AdamW optimizer [45], with learning rate 5e-5 and weight decay 1e-5. We apply linear learning rate warmup in the first two epochs followed by cosine learning rate scheduling that eventually decays the initial learning rate (after warmup) by 10x. We additionally clip the gradient norms at 5.0 as we empirically find this helps with learning. We train our model for 100 epochs on the Highway dataset and 40 epochs on the AV2 dataset with batch size of 4. Please see supp. for more details on the point encoder and attention network.
Baselines: We compare our proposed LabelFormer with state-of-the the-ttt auto-labelling methods Auto4D [1] and 3DAL [2]. Since neither papers released their code, we reimplemented both methods based on their implementation details and thoroughly tuned the hyperparameters. We follow the original implementations to set the Auto4D's temporal window size to be 10. For 3DAL dynamic branch, we the original bounding box temporal window size of 101 and raised the point window size from 5 to 11 to increase model performance. Note that since the Highway dataset contains very

---

Dataset
First-Stage Detector
Second-Stage Refinement
Mean IoU
RC @ 0.5
RC @ 0.6
RC @ 0.7
RC @ 0.8
62.60
76.39
65.72
48.66
25,04
PointPillars [37]
A 11|
65.49
79.47
70.35
56.04
32.40
3DAL [2]
64.58
77.25
Ours (LabelFormer)
68.28
81.22
73.22
60.68
40.78
AV2
65,70
78.92
68.90
54.37
32.44
Auto4D [i]
67.92
81.73
73.14
59.42
37.02
VoxeINeXt [46]
3DAL [2]
67.12
80.36
71.42
57.83
36.01
Ous (LabelFormer)
70.18
83.26
75.06
63.04
43.76
60,20
69.83
59.62
46.11
26.58
PointPillars [37]
AutoAD [1]
65.63
77.01
68.23
56.28
37.08
3DAL [2]
66,03
79.44
70.33
56.62
35.18
Highway
Ours (labeeFormer)
70,59
83.09
75.77
65.11
46.16
65.90
76.76
68.32
56.27
36.77
VoxeINeXi [46]
Auto4D [1]
69.27
79.36
72.81
3DAL [2]
68.17
80.61
73.07
60.61
38.85
Ours (LabelForner)
72.38
83.68
77.45
68.33
50.06
Table I: Comparison with state-of-the-art
few static objects, we train the 3DAL dynamic branch with all objects in the scene and only apply the dynamic network during inference. For the AV2 dataset we apply the original 3DAL method with both stationary and dynamic branches.
First-stage initialization: We obtain the first-stage coarse initializations by running a detection and tracking model. For fair comparison between the refinement approaches, we train and evaluate all refinement models on the same set of true-positive first-stage object trajectories, ie., those that can be associated with a ground-truth object trajectory (more details supp.). To ensure our con- clusions generalize, for each dataset, we evaluate the refinement approaches on initializations from two detection models. Specifically, we experiment with a multi-frame version of PointPlllars [37] and recent state-of-the-an detector VoxelNeXt [46] as the first-stage detector. Following [1, 2}, implement simple rule-based multi-object tracker, please see supp. for more details.
Refinement results against SOTA: Table
1o0 wawe seuuiee
12
Gun
AV2 Mean leU (8)
70
burs
shows that in terms of refinement accuracy, our method consistently outperforms the initializa- tions and state-of-the-arr auto-label refinement methods by a large margin across both detector initializations on both datasets. While existing methods already have significant gains over the initializations, our method is able achieve on average 92% more mean IoU gains. In addition, accuracy on both static and dynamic objects with single network, as shown in supp. Furthermore, we measure the average refinement run time on trajectories from the VoxeINeXt initialization, using a single RTX2080 GPU. Results are shown in Fig. 3 alongside mean IoU, We find that LabelFormer 2.7x faster than the window-bases dynamic 3DAL on the Highway dataset. On AV2, which has 52% static objects, our method still slightly faster than 3DAL which uses non- non-window-bart static branch. Auto4D uses all available points while 3DAL samples at most 1024 points per frame, resulting in Auto4D having much longer time but higher performance large IoU thresholds.
71
69
70
Autodp
68
AutorD
2DAL
2DAL
60
10'
10*
102
10
Inference time (ms) par traj
Inforence time (ms) per tra)
Figure Refinement quality vs. runtime
method is able achieve significantly higher
Improving object detection with auto-labeled data:
Aute-Label
Mean AP
APG005
AP6:0.7
AP6:0.8
We additionally study the effect of the refined auto- labels in the downstream object detection task. Specif- ically, we use VoxelNeXt [46] as the first-stage de- tector and train various auto-labellers on the main Highway training set (consisted of 150 human-labelled snippets), and use them to label an additional 500 Highway snippets. We then train a downstream object detector with combined dataset of 1,0 numan-labelled snippets and 500 auto-labelled snip- pets. Table 2 shows the average precision (AP) results. Overall, training with the bigger dataset augmented with luto-labels is better than with the labelela dataset alone, and our refined auto-labels give the biggest boost with 3% gain 0.8 IoU.
NIA
82.98
91.62
79.17
55.97
83.63
SDAL
83.64
92.56
79.75
55.79
Ours
84.81
92.91
80.91
59,00
Table 2: [Highway] Downstream task
7

---

Box Enc.
Point Enc.
Perturb
Window
# AlL
Mean IoU
RC@0.5
RC
RC
RC@00.8
M,
All
69.20
80.68
73.92
63.28
43.76
All
70.17
45.61
Ms
All
70.97
82.37
75.37
65.32
47.65
M4
All
71.59
83.62
76.35
66.62
48.87
Ms
AlI
72.11
83,93
77.50
67.66
49.90
M6
V
S
71.18
82.33
75
65.81
48.80
10
71.72
83.26
76.57
66.68
49.69
Mss
20
71.92
83.14
76.62
67.08
49.91
My
V
All
72.38
83.68
77.45
68.33
50,06
Table 3: [Highway] Ablation study using the VoxelNeXt initializations. Perturb refers to the bounding box perturbation augmentation, Window specifies the window size when applicable, and "# Att" is the number of self-attentio blocks.
Initialization
LabelFormer
Figure 4: Qualitative results:
Initialization vs. refinement. Auto-labels in orange, GT in magenta.
Effect of box and point features: Mi and M2 in Table each only encode box features and point features respectively. Comparing to Mg which uses both features, we show that both the box and point features in the per-frame encoder stage contribute to the overall success of the model.
Effect of per-frame perturbation: M3 + Mg in Table shows that the per-frame bounding box perturbation augmentation use helps with the final performance.
Effect of number of self-attention blocks: M4. Ms and My in Table 3 show that the refinement accuracy grows with more attention blocks.
Effeet temporal context length: Finally, we train and run our method a window-based fashion to restrict the temporal context given each method. Ms. M7, Mg and Mg in Table 3 show that the refinement accuracy steadily increases with more temporal context given the model.
Qualitative results: Fig. 4 shows an example for which the observations are very sparse at the be- ginning of the trajectory and get denser afterwards. LabelFormer is able to exploit the full temporal context and improve the bounding box trajectory, especially at the most challenging frames. More examples and comparisons with previously proposed refinement methods can be found in the supp.
Limitations
The two-stage auto-labelling paradigm has an inherent limitation: the second stage only refines the continuous bounding box localization errors, but does not correct discrete detection errors (false positives, false negatives) and tracking errors (id switches, fragmented tracklets). Such discrete errors will propagate to the final output. Moreover, the refinement performance on the fragmented tracklets may be sub-optimal due to the missing temporal context (otherwise present in the un- fragmented tracklets). Therefore, a future direction to explore alternative paradigms that can recover from discrete errors too. Finally, failure mode our proposed refinement model is that can degrade the quality of the auto-labels with respect to initialization when the input trajectories are short and have sparse observations. Such cases challenging try to address this by estimating auto-label uncertainty and leveraging it in downstream applications.
humans, yet future work
6 Conclusion
In this work, we study the trajectory refinement problem in two-stage LiDAR-based offboard perception paradigm. Our proposed method, LabelFormer, is a single transformer based model that leverages full temporal context of the trajectory. fusing information from the initial path as well as the LiDAR observations. Compared to prior works, our method simple, achieves much higher accuracy and runs faster in both urban and highway domains. With the ability to auto-label larger dataset effectively and efficiently, LabelFormer helps boost downstream perception performance, and unleashes the possibility for better autonomy systems.

---

Acknowledgments
We thank the anonymous reviewers for the helpful comments and suggestions. We would also like to thank Bin Yang and loan Andrei Bårsan for insightful discussions and guidance at the early stage of the project. Finally, we would like to thank the Waabi team for their valuable support.
References
[1] B. Yang, M. Bai, M. Liang, W. Zeng, and R. Urtasun. Auto4d: Learning to label 4d objects from sequential point clouds. CoRR, abs/2101.06586, 2021. URL bettp://aaxxiv.rgg abs/2101 2185210666686.. 06586.
[2] C. R. Qi, Y. Zhou, M. Najibi, Sun, K. Vo, B. Deng, and D. Anguelov. Offboard 3d object de- tection from point cloud sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 61 34-6144, June 2021.
[3] B. Wilson, Qi, T. Agarwal, Lambert, Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hart- nett, J. Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[4] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference computer vision and patler recognition, pages 2020.
[5] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://prroceedingg neurips. wwwpurissccppppppp...e #tee:/38776ee11100sccccacaaaaaa .df..
[6] Zakharov, W. Kehl, Bhargava, and A. Gaidon. Autolabeling 3d objects with differentiable rendering of sdf shape priors. In IEEE Computer Vision and Pattern Recognition (CVPR), June 2020.
[7] 7. Pang, Z. Li, and N. Wang. Model-free vehicle tracking and state estimation in point cloud sequences. IROS, 2021.
[8] J. Ye, Y. Chen, N. Wang, and X. Wang. Online adaptation for implicit object tracking and shape reconstruction in the wild. IEEE Robotics and Automation Letters, 2022.
[9] Z. Qin, J. Wang, and Y. Lu. Weakly supervised 3d object detection from point clouds. In Pro- ceedings of the 28th ACM International Conference on Multimedia, MM '20, page 4144 4152, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978145033 458179987 doi: 1i::0.1145/3394177 111.341385.. URL https::pp... / /oii..... org/ /0////1...0 11111/339/31111... 11444//3394411134188855
[10] P. Pfreundschuh, H. F. Hendrikx, V. Reijgwart, R. Dubé, R. Siegwart, and A. Cramariuc, Dy- namic object aware lidar slam based on automatic generation training data. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11641-11647, 2021. doi: 10.1109//CA44506.66 202. 1.960703..
[11] X. Chen, B. Mersch, L. Nunes, R. Marcuzzi, I. Vizzo, J. Behley, and C. Stachniss. Automatic labeling generate training data for online lidar-based moving object segmentation. IEEE Robotics and Automation Letters, 7(3):6107-61 14, 2022. 10.: (09/LRRA.002..166644
[12] Y. You, K. Luo, C. Phoo, W.-L. Chao, W. Sun, B. Hariharan, M. Campbell, and K. Q. Weinberger. Learning to detect mobile objects from lidar scans without labels. In CVPR, 2022.

---

[13] L. Zhang, A. J. Yang, Y. Xiong. S. Casas, B. Yang, M. Ren, and R. Urtasun. Towards unsuper- vised object detection from lidar point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2317-9328, June 2023.
[14] L. Fan, Y. Yang, Y. Mao, F. Wang. Y. Chen, N. Wang, and Z. Zhang. Once detected, never lost: Surpassing human performance in offline lidar based 3d object detection. In ICCV, 2023.
[15] Ma, X. Yang, H. Zhou, X. Li, B. Shi, J. Liu, Yang. Liu, I.. He, Y. Qiao, Y. Li, and Li. Detzero: Rethinking offboard 3d object detection with long-term sequential point clouds. In Proceedings of International Conference on Computer Vision (ICCV), 2023.
[16] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048800 2018.
[17] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,,022 2020.
[18] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[19] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and Schmid. Vivit: video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836-6846, 2021.
[20] T. Meinhardt, A. Kirillov, L. Leal-Taixe, and C. Feichtenhofer. Trackformer: Multi-object tracking with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8844 8854, 2022.
[21] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259-16268, 2021.
[22] D. Wang, X. Cui, X. Chen, Z. Zou, T. Shi, S. Salcudean, Z. J. Wang, and R. Ward. Multi-view 3d reconstruction with transformers. In Proceedings of the IEEF/CVF International Confer- ence on Computer Vision, pages 5722-5731, 2021.
[23] Lu, D. Batra, Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic repre- sentations for lannluuggee tasks. Advances in neural information processing systems, 32, 2019.
[24] A. Prakash, K. Chitta, and A. Geiger. Multi-modal fusion transformer for end-to-end au- tonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pat- tern Recognition, pages 7077-7087, 2021.
[25] C.-Y. Wu and P. Krahenbuhl. Towards long-form video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1884-1894, 2021.
[26] B. Ni, H. Peng, M. Chen, S. Zhang, G. Meng, J. Fu, S. Xiang, and H. Ling. Expanding language-image pretrained models for general video recognition. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Purt IV, pages 1-18. Springer, 2022.
[27] N. Dvomikk I. Hadji, R. Zhang, K. G. Derpanis, R. Wildes, and Jepson. Stepformer: Self-supervised step discovery and localization in instructional videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Patter Recognition, pages 18952-18961, 2023.
10

---

[28] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, et al. Tnt: Target-driven trajectory prediction. In Conference on Robot Learning, pages 895- 904. PMLR, 2021.
[29] A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, V. Badrinarayanan, R. Cipolla, and A. Kendall. Fiery: future instance prediction in bird's-eye view from surround monocular cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15273-15282, 2021.
[30] A. Cui, Casas, K. Wong, S. Suo, and R. Urtasun. Gorela: Go relative for viewpoint- newwpint-innariiat motion forecasting. arXiv preprint arXiv:221 2022.
[31] R. Mahjourian, J. Kim, Y. Chai, M. Tan, B. Sapp. and D. Anguelov. Occupancy flow fields for motion forecasting autonomous driving. IEEE Robotics and Automation Letters, 7(2): 5639-5646, 2022.
[32] B. Agro, Q. Sykora, S. Casas, and R. Urtasun. Implicit occupancy flow fields for perception and prediction in self-driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1379-13888 2023.
[33] H. Fan, F. Zhu, C. Liu, L. Zhang, L. Zhuang, D. Li, W. Zhu, J. Hu, H. Li, and Q. Kong. Baidu apollo em motion planner. arXiv preprini arXiv: 1 8807008048, 2018.
[34] A. Sadat, M. Ren, A. Pokrovsky, Y.-C. Lin, Yumer, and R. Urtasun. Jointly learnable behavior and trajectory planning for self-driving vehicles. In 2019 IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3949-3956. IEEE, 2019.
[35] S. Casas, A. Sadat, and R. Urtasun. Mp3: A unified model to map, perceive, predict and plan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14403-14412, 2021.
[36] K. Renz, K. Chitta, O.-B. Mercea, A. Koepke, Z. Akata, and A. Geiger. Plant: Explain- able planning transformers via object-level representations. arXiv preprint 1rXiv:2210.1477 14.222 2022.
[37] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast en- coders for object detection from point clouds. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12697-12705. 2019.
[38] C. R. Qi H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d clas- sification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
[39] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[40] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
[41] Besl and N. D. McKay. A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):23-2266, 1992. doi:10.1100/34, 1211991.
[42] J. Park, Q.-Y. Zhou, and V. Koltun. Colored point cloud registration revisited. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 143-152, 2017. doi:10. 109/ ICCV.2017.25,
[43] J. L. Ba. J. R. Kiros, and G. E. Hinton. Layer normalization. 2016.

---

[44] O. Press, N. Smith, and M. Lewis. Train short. test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022.
URL
[45] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Con- ference on Learning Representations, 2019. URL https://openreview.nee Bkg6RiCqY7.
[46] Y. Chen, J. Liu, X. Zhang, X. Qi, and J. Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
[47] X. Weng, Wang, D. Held, and K. Kitani, 3D Multi-Object Tracking: Baseline and New Evaluation Metries, IROS, 2020.
12

---

Supplementary Materials
A Implementation and Experiment Details
A.1 Model Details
The point encoder is consisted of a voxelizer followed by CNN-based backbone and Feature Pyramid Network (FPN) [40].
Point Encoder:
Specifically, the voxelizer employs voxel resolution of 10em in all of X, and Z directions, with a region of interest of [-12, 12] meters along X, [-4, 4] meters along Y, and [-0.2, 3.0] meters along Z, to construct a N₂ Ny N₂ voxel grid with N 240, N2 = 80 and N₂ 320. For all points in each 3D voxel grid, we first represent each point (Ar, Ay. Az, At) where (Ax, Ay, Az) is the positional offset with respect to the voxel centroid and AV = bref is the difference between the per-point time and the LiDAR sweep end time of the middle frame in the object trajectory. We feed this four-vector representation of each point into a two-layer MLP with 16 output channels each, and LayerNorm [43] and ReLU applied right after the first layer. Then, for each voxel, we pool all point features inside by summing them and applying LayerNorm after to derive voxel grid features NR.N,xN,xx. NNN,, x16 which can be viewed N₂ x N, "feature pillars" along the Z axis. We additionally encode z-axis positional embedding via learnable variable block E R 1 We concatenate the non-empty voxels in each feature pillar with the positional embedding block to obtain an augmented feature E IR NR, x32 (N₂ S N₂ is the number of non-empty voxels in the pillar), and pass through two-layer MLP with 16 and 32 output channels each (with LayerNorm and ReLU in between), apply LayerNorm after the second layer, and sum all the features along each pillar to obtain a BEV feature mapE N. NN,,N, x32
The backbone takes the BEV feature map as input and first applied three stem layers with 120, 96, 96 output channels each. Each stem layer is consisted of a 3x3 convolutional layer, followed by GroupNorm (GN) and ReLU. The first stem layer has stride of 2 while the next two have stride of 1. Then, the output passes through three downsampling stages. The three downsampling stages contain 6, ResNet [39] blocks with 288, 384, 576 output channels respectively. Each ResNet block applies sequence of 1xl conv, GN, ReLU, 3x3 conv (with an optional stride parameter), GN, ReLU, 1x1 conv, GN, ReLU to obtain residual and sum with the input. Fach downsampling stage downsamples the input by a factor of 2x within the first ResNet block, where the first ResNet block has stride 2 in the middle 3x3 conv, and the residual output is added to the input that is downsampled with a Ixl conv block with stride 2 followed by GN. The remaining ResNet blocks in each downsampling stage all have stride of 1.
The FPN takes the outputs from all three stages in the backbone, which are 4x, 8x and 16x down- sampled from the original resolution (the stem layers downsample the input by 2x, and each down- sampling stage in the backbone further downsamples by 2x). The FPN module fuses the two lowest resolution feature maps first by applying a 1x1 cony block + GN to the 16x low-resolution map, up- sampling it by 2x with bilinear interpolation, and adding it to the 8x downsampled feature map. We then perform a similar operation to fuse the 4x downsampled feature map with the newly fused 8x downsampled feature map, and apply final 3x3 conv output feature map of 4x downsampled original resolution with channel dimension 256 the feature map of per-frame points.
Attention Block: We next provide more details on the feed-forward MLP in each attention block. The feed-forward MI.P is consisted of linear layer with input dimension 256 and output dimension 512, followed by ReLU, DropOut with 10%, second linear layer with input dimension 512 and output dimension 256 and another DropOut with 10%. We add the output of the MLP to the input of the MLP and return the sum.
13

---

Training Loss Details: In this section, we provide detailed definitions for our loss functions. The regression loss is defined as:
À
Lreg((bi),,bb) (b;}) = Σ
M (01,th) + smooth/1 (9:. v()) smooth(](1,"") + smoothl (1)
1
smoothe; (sin (20.), sin (207)) + smoothe (2:s();;), cos (20;))
M
(4)
with the hyperparameter À 0.1 in practice, and the loU loss is given by:
Liou (b;}) = M E
(BBoxx,, ūt, u),, BBox(2) vi,1" w²))
(5)
to compare the axis-aligned refined and ground-truth bounding box in each frame.
A.2 Detector and Tracker
To obtain the first-stage coarse initialization, we follow the standard "detect-then-tree" thenntrack" approach where detection model trained to output per-frame detections and leverage a tracker obtain consistent tracklets over time. Next, give more details about the detector and tracker use.
Detector: To boost detection performance, we adapt the single-frame public implementation of both PointPillars [37] and VoxelNeXt [46] multi-frame version that additionally takes 4 past history frames and 4 future frames input. The validation mean AP of the single-frame frame multi- frame PointPillars models are 68. 78% and 71.02% on the Highway dataset respectively, and 55.98% and 60.58% on AV2 respectively. The validation mean AP of hingle-frame vs. multi-frame Voxel- NeXt are 31.87% on Highway and 60.06% on AV2.
Tracker: Following [1, 2], we use simple online tracker, which is largely inspired by [47], and we provide the implementation details of our tracker, in particular how association is performed across frames.
For each new frame at time step t with detections B, = {b/} where each bj = (=/, of. 44 w/, 8.) E RR.5 is the individual 2D BEV bounding box, we first filter with Non-Maximum Suppression with IoU threshold 0. and then filter out bounding boxes with low confidence scores. We then compute cost matrix with existing tracklets S, {s/} } follows. For each tracklet j. first predict its bbox position (z), vt) at time t: if the tracklet has at least two past frames, we set (xt. v() 2 (12-1198/1)) - (x/-2, vi-2) via naive extrapolation (assuming constant velocity between two adjacent frames); otherwise we simply set (xt v/) ( Then, for each pair of the detected bbox by and the predicted tracklet bbox b₁. we compute the Euclidean distance between the bbox centroids gi. V 2²)² + (uf 2 For each existing tracklet, simply employ greedy strategy to find the nearest detection /* arg min; Gid, and if the closest distance pird" is greater than a threshold of 5.0m, then the tracklet has no match. We use greedy matching instead of a more sophisticated matching strategy such as Hungarian matching because it is more robust to noisy and spurious detections.
If a tracklet is matched to a new detection, we add the detection to the tracklet and update the tracklet score cf
v. tc where e/_ is the old tracklet score, d = 1.0 is the detection confidence score w+10
we set for every new detection, and w the tracklet.
0.9* where n²!_ is the number of tracking steps in
If tracklet is not matched, we grow the tracklet by naively extrapolating the position and angle, and set the new confidence score of 0.9c/-1-
If a new detection is not matched to any tracklet, we start new tracklet and initialize the confidence score ei with the detection's 's confidence.
14

---

Motion State
First Stuge-Suge Delector
Secund Stage Relinement
Mean IoU
RC @ 0.5
RC @i 0.6
RC @ 0.7
RC 0.8
64.46
79.34
69.95
53.58
28.66
PointPillars
Auto4D
67.51
82.56
74.66
61.26
36.91
3DAL
68.00
81.35
75.04
63.26
41,91
Stationary
Ours (LabelFormer)
70.67
84.08
77.31
66.03
46.81
68.91
83.53
75.15
61.53
38.81
Auto4D
VoxeINeXt
3DAL
70.63
84.55
77.74
66.50
45.13
Ours (LabelFormmer)
73.21
86.77
80.09
69.54
51.08
60.53
73.12
61.04
43.23
21.03
Auto4D
PointPillars
3DAL
60.80
72.70
60.03
45.36
22.12
Dynamic
Ours (LabelFormer)
65.64
78.06
54.74
34.10
62.25
73.96
62.17
46.66
25.57
VoxeINeXt
Auto4D)
64.73
77.01
66.77
52.02
30.72
3DAL
63.33
75.85
64.60
48.55
26.
(labelFormer)
Table 4: [AV2] Performance break-down for ground-truth stationary vs. dynamic objects
Bbox Enc.
Point Ene.
Perturb
Window
# All
Mean IoL
RC@00.5
RC@0066
RC#007
RC@008
Mi Ma Ma Ma
All
3
64.20
69.83
59.62
46.11
26.58
All
3
68.91
80.84
73.05
61.65
42.89
74.17
Mo
10
3
69.33
81.83
74,80
64.12
45.42
Me
3
70.59
83.09
75.77
65.11
46.16
All
6
70.93
83.50
76.51
66.34
47.85
Table 5: [Highway] Ablation study using the PointPillars initializations. Perturb refers to the bounding box perturbation augmentation, Window specifies the window size when applicable, and # Att is the number of self-attention blocks.
We terminate all tracklets with tracking confidence score less than 0.1, and apply NMS at the end over all existing tracklets in the current frame with an IoU threshold of 0.1. We repeat this process for the next frame at time + 1 until the end of the sequence.
A.3 Association with GT Trajectories
For each initial object trajectory detected and tracked in the first stage, we use simple heuristic to associate it with ground-truth truth object trajectory as follows: for each frame that the detected trajectory is present, we identify the ground-truth bounding box that has the maximum IoU with the detected bounding box in that frame. such ground-truth truth box has loU less than 10%, then we fail to find a matching ground-truth box this frame. As a result we obtain M' ground-truth object IDs for detected trajectory of length M, with < M' .M as we might not be able find ground-truth ID for every frame. If M' is then we have failed to find associated ground-truth object: consider the detected object as false positive and discard in trajectory refinement training and evaluation. Otherwise we take the most common ground-truth actor id out of the M' objects and assign as the associated ground-truth object trajectory for training and evaluation.
B Additional Experiments
Static vs. Dynamic Objects The AV2 validation set contains around 52% stationary objects (we classify an actor as static the max displacement in the ground-truth displacement in all X, and z direction is within 1.0m). Table 4 additionally shows the dynamic vs. stationary object break-down on the AV2 dataset. Our method is able to achieve significantly higher refinement accuracy on both static and dynamic objects with single network.
We additionally performed the same set of ablation studies as Table 3 in the main paper on the Highway dataset with the PointPillars-based initializations. Table shows the results, which give the same conclusions the VoxeINeXt-baal initializations in the main paper.
Ablation with PointPillars Init:
15

---

Archilecture
Mean IoU
RC @i 0.5
RC @ 0.6
RC @ 0.7
RC @ 0.8
Init
65.90
76.76
68.32
$6.27
36.77
MLP MLP((-layer (1-layer)
70.14
81.56
74.20
63.95
46.23
MLPP (3.layer)
70.50
82.31
74.82
65.04
46.80
MLP (6-layer)
70.50
82.31
74.82
65.04
46.80
LahelForerr(s block) [44]
Table 6: [Highway] Ablation of MLP Transformer using the VoxelNeXt initializations
Positional Encoding
Mean IoU
RC 0.5
RC U.6
RC @ 0.7
RC @ 0.8
Absolute [5]
71.71
83.63
76.97 77.45
67.11 68.33
48.67
50,06
Table 7: [Highway] Ablation of positional encoding using the VoxelNeXt initializations
MLP vs. Transformer To understand whether attention/transformrriike architecture helps, we ablate the effect of the cross-frame attention module by replacing with an MLP. Specifically, to aggregate features across frames, we mean pool the per-frame features from all frames, apply an MLP to the pooled features, and then add the aggregated feature back to each frame-level feature. The updated per-frame features are then passed to the decoder module of LabelFormer. For the MLP, each layer consists of linear layer, followed by LayerNorm and ReLU. We conducted this experiment with 1, 3, 6 layers respectively. The results in Table 6 show that the cross-frame attention module outperforms the MLP architecture by a large margin, demonstrating the benefits of attention with 40.9% higher relative gain in mean IoU.
Positional Encoding We additionally ablate our choice of positional encoding with the VoxelNeXt initialization on the Highway dataset. Table 7 shows that the relative positional encoding AliBi [44] gives overall better performance than using the vanilla absolute positional encodings [5].
C Qualitative Results
In this section we show qualitative results for trajectory refinement, comparing LabelFormer with the coarse initialization, 3DAL [2] and Auto4D [1].
We illustrate initial and refined auto-labels for the Highway dataset with VoxeINeXt initializa- tions. Fig. showcases trajectories of two objects on the top that have sparse observations (and hence worse initializations) at the beginning, and denser observations towards the end, and ours La- belFormer is able to give better refinement for the worse initializations because it is able to leverage more temporal context more effectively than previous works. Fig. 5, 6, 7, 8 additionally showcase that our method works better qualitatively on trajectories with both sparse and dense observations and with various speeds. For more visualizations on Argoverse, please refer to the supplementary video.
16

---

IaCHan MA 7322
TUTT
SDAL Neantol: 77.25
Labe Former Neen 89.34
SDAL Kean lol:: 80.71
Aute4D Mear InU: 82.59
Figure 5: [Highway] Qualitative results showcasing different object trajectories (first-stage init, refined by 3DAL, Auto4D and Ours LabelFormer) in each object's trajectory coordinate frame. The ground-truth bounding box is in magenta, and the auto-label is in orange. To avoid cluttering, we visualize every other three bounding box in the first 50 frames of the trajectory.
17

---

BWN
Jube BLSS
IDAL Moan WU: 67.26
Labeloomer Me tet: $1.50
Init Mean loU: 72.12
3DAL Mean loU: 81.15
Auto4D Mean loU: 84.38
LabelFormer Mean loU: 85.98
Figure 6: [Highway] More qualitative results. Ground-truth in magenta, auto- in orange.

---

Nean $2.84
30AL
Autort 61
Labefomer Maan
Init Mean toU: 63.06
3DAL Mean loU:
Auto4D Mean loU: 74.44
LabelFormer Mean loU: 85.72
Figure 7: [Highway] More qualitative results. Ground-truth in magenta, auto-labels in orange.
19

---

inte lol: $3.74
82.22
Laselfomer Hear (00) 82.37
Auu40 Moan $153
Mase B4
Figure 8: [Highway] More qualitative results. Ground-trut in magenta, auto-labels in orange.
20


Reviews:
[{"review_id": "w2xac_vaQ1", "reviewer": "Reviewer_pNiA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a trajectory-level refinement approach for auto-labelling objects in LiDAR point clouds. The approach uses self-attention for reasoning about trajectories, and decodes refined object size and per-frame poses. Evaluation are performed on urban and highway datasets.", "summary_of_recommendation": "The studied problem is not a valid problem to study. It is not related to robot learning either.\n\n---\n\nAfter rebuttal, the authors addressed the most important concerns of \"how the auto annotated is used/evaluated\". So I raised my ratings.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The authors can respond to the comments in the review.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The whole problem formulation of automatic labeling has a serious problem or paradox: \n  - If this automatic labeling method is so good and can generate nearly perfect object annotations, this problem should have already been considered solved. If that's the case, why not just directly deploy the automatic labeling method in the applications? Why are you still waste your time to annotate data with this method considering the problem has been solved?\n  - If this automatic labeling method is not so good and cannot generate nearly perfect annotations, why can this method be used to generate annotations which is supposed to be ground truth and be 100% accurate?\n\nIt is true that there have been some previous works published in relevant robotics and/or computer vision venues such as Auto4D that call themselves \"automatic labeling\", it does not justify that automatic labeling is a valid problem to research. I would argue that methods that study the problem of \"automatic labeling\" such as Auto4D should never be published.\n\nIt would be a valid problem if the authors can call their method \"a method for 3D LiDAR point cloud object tracking\", \"a method for 3D LiDAR point cloud object pose estimation\" etc.\n\n---\n\nAfter rebuttal, the authors addressed the most important concerns of \"how the auto annotated is used/evaluated\". So I raised my ratings.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "WGGTtMy5bk", "reviewer": "Reviewer_xWt1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents a trajectory refinement method that is based on detection and tracking of individual objects.  LIDAR automatic labeling across frames allows for augmentation and each sequence is modeled via transformer.  The initial representations are the combination (per frame) of both bounding box and point clouds.  The frames are then decoded (via MLP)  providing a correction. ", "summary_of_recommendation": "The approach is intuitive and the results convincing but why the model is helpful is unclear and therefore gives me pause about where the innovation really lies.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I thank the authors for the inclusion of the ablation studies as the design choices are not immediately obvious from the setting.  That being said, it's not clear the transformer is either as effective as the authors believe or that it's effective for the reasons we want.\n\n1. Given that a single attention layer is nearly equivalent in performance to much more expensive settings (aka 6 layers), do we need attention? Or do we think other factors like the normalization are what actually matters?\n2. A similar question can be raised about the effect of the window size.  \n\nA simple comparison would be to train a CNN or MLP over a small context window and perform the same training.  \n\nMinor: I don't see a discussion of how many heads were used in the attention blocks", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The results show noticeable improvements across settings and metrics.  The primary question that's unclear to me is the actual effect of the model choices.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "1BjNhX7HIE", "reviewer": "Reviewer_b8wo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a transformer-based approach called LabelFormer for trajectory refinement to automatically label objects in LiDAR point clouds. The proposed method aims to refine the bounding box trajectory given a noisy initialization obtained from a detect-then-track paradigm. The paper presents multiple experimental results demonstrating the superiority of LabelFormer compared to existing methods in terms of accuracy, efficiency, and downstream detection performance.", "summary_of_recommendation": "I would give a weakly accept. The proposed approach is effective and efficient in addressing the trajectory refinement problem. The paper provides a clear presentation of the method and comprehensive evaluation results. However, additional discussion compared to existing methods, and more in-depth ablation studies would further enhance the quality of the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "1. What are the differences between this work from other related studies?\n2. Is there any way to enable the proposed trajectory refinement method without relying on ground truth bounding box labels?\n3. What are the outcomes when employing suboptimal object detection and tracking methods?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n1. This submission introduces a novel transformer-based architecture for trajectory refinement, which effectively handles both static and dynamic objects, resulting in improved accuracy and efficiency.\n2. The paper provides a thorough experimental evaluation on real-world datasets, showing the superiority of LabelFormer over existing methods for trajectory refinement. It also demonstrates the positive impact of using LabelFormer's auto-labels for training downstream object detectors.\n3. The paper is generally easy to follow and well organized.\n\nSuggestions:\n\n1. Several related works [44, 45, 46] are missing in the current discussion. Those methods are very related to the proposed methods and therefore should be well compared and discussed.\n[44] Pfreundschuh et al., Dynamic object aware lidar slam based on automatic generation of training data, ICRA 2021\n[45] Chen et al., Automatic labeling to generate training data for online LiDAR-based moving object segmentation, RAL 2022\n[46] Jiang et al., Improving the intra-class long-tail in 3d detection via rare example mining, ECCV 2022\n\n2. As mentioned, this submission mainly focuses on bounding box refinement rather than direct label generation. Furthermore, the proposed transformer-based network still relies on ground truth bounding boxes for supervision. Therefore, I would recommend considering a title that more accurately reflects the focus on trajectory refinement rather than auto-labeling.\n\n3. As outlined in the limitation section, the effectiveness of the proposed method can be significantly affected by the performance of the initial object detection and tracking stage. However, there is currently no clear solution to address the potential consequences of using suboptimal object detection and tracking methods, which may impact the overall results.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a trajectory-level refinement approach for auto-labeling objects in LiDAR point clouds. The reviewers find the paper to be well written and appreciate the thorough experimental evaluations. They raised several different concerns which have been satisfactorily addressed in the rebuttal. Although this paper does not score significantly high in the impact, it is expected to be of interest to researchers in the field.
Decision: Accept (Poster)