RETR: Multi-View Radar Detection Transformer for Indoor Perception
Authors: Ryoma Yataka
Adriano Cardace
Pu Perry Wang
Petros Boufounos
Ryuhei Takahashi
Venue: NeurIPS 2024 poster (2024)
Keywords: indoor monitoring
radar perception
object detection
instance segmentation
detection transformer
multi-view radar heatmaps
Primary area: machine_vision
TL;DR: We propose RETR for multi-view radar indoor perception, improving object detection and segmentation, outperforming current methods on indoor radar datasets.

Abstract: Indoor radar perception has seen rising interest due to affordable costs driven by emerging automotive imaging radar developments and the benefits of reduced privacy concerns and reliability under hazardous conditions (e.g., fire and smoke). However, existing radar perception pipelines fail to account for distinctive characteristics of the multi-view radar setting. In this paper, we propose Radar dEtection TRansformer (RETR), an extension of the popular DETR architecture, tailored for multi-view radar perception. RETR inherits the advantages of DETR, eliminating the need for hand-crafted components for object detection and segmentation in the image plane. More importantly, RETR incorporates carefully designed modifications such as 1) depth-prioritized feature similarity via a tunable positional encoding (TPE); 2) a tri-plane loss from both radar and camera coordinates; and 3) a learnable radar-to-camera transformation via reparameterization, to account for the unique multi-view radar setting. Evaluated on two indoor radar perception datasets, our approach outperforms existing state-of-the-art methods by a margin of 15.38+ AP for object detection and 11.91+ IoU for instance segmentation, respectively. Our implementation is available at https://github.com/merlresearch/radar-detection-transformer.

RETR: Multi-View Radar Detection Transformer for Indoor Perception
Ryoma Vatakal.3,+. Adriano Cardace Pu (Perry) Wang".", Petros Boufounos', Ryuhei Takahashi Mitsubishi Electric Research Laboratories (MERL). USA Department of Computer Science and Engineering, University of Bologna, Italy Information Technology R&D Center (ITC), Mitsubishi Electric Corporation, Japan
Abstract
Indoor radar perception has rising interest due to affordable costs driven by emerging automotive imaging radar developments and the benefits of reduced privacy concerns and reliability under hazardous conditions (e.g., fire and smoke). However, existing radar perception pipelines fail to account for distinctive char- acteristics of the multi-view radar setting. In this paper, propose Radar dE- tection TRansformer (RETR), an extension of the popular DETR architecture, tailored for multi-view radar perception. RETR inherits the advantages of DETR, eliminating the need for hand-crafted components for object detection and seg- mentation the image plane. More importantly, RETR incorporates carefully designed modifications such as 1) depth-prioritized feature similarity via tun- able positional encoding (TPE); 2) a tri-plane loss from both radar and cam- era coordinates; and 3) a learnable radar-to-camera transformation via reparam- eterization, account for the unique multi-view radar setting. Evaluated on two indoor radar perception datasets, approach outperforms existing state- of-the-art methods by a margin of 15.38+ AP for object detection and 11.91+ IoU for instance segmentation, respectively. Our implementation is available at trans ormer.
https: giggghtt.b..... htt.ah/...o/eeerreeeeaaaaaaaaaaa
Introduction
Perception information encompasses the processes and technologies to detect, interpret, and under- stand their surroundings. Complementary to the mainstream camera and LiDAR sensors, radar can enhance the safety and resilience of perception under low light, adversarial weather (c.g., rain, snow, dust), and hazardous conditions (c.g., smoke, fire) at affordable device and maintenance cost. An emerging application radar perception indoor sensing and monitoring for elderly care, building energy management, and indoor navigation [7]. A notable limitation of indoor radar perception is the low semantic features from radar signals.
Earlier efforts use radar detection points [42, 30] to support simple classification tasks such as fall detection and activity recognition over limited number of patterns. To support challenging perception tasks such as object detection, pose estimation, and segmentation, lower-level radar signal representation such as radar heatmaps is more preferred. Along this line, the earliest work RF-Pose [43] using convolution- -bsee autoencoder network to fuse features from the two radar views and regress keypoints for 2D image- maagee pose estimation. is later extended to 3D human
Equull contribution.
'The work was done as a visiting scientist ITC Mitsubishi Electric Corporation.
'The work was done during his internship at MERL.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

RETR: Radar dEtection TRansforeer
Imege Flane
Hodronal Mersio
Hortzonall View
Deepth
Ueceenre R (rrccd sereenaaa
Transfomer Ennodor Dananer
(10)))
Horizantal Ver ukcal
Set of obsect querles wilh sparia embedd ngs
Figure 1: By taking Norizontallview and vertical-view radar heatmaps inputs, RETR introduces depth-prioritiznng positional encoding (exploit the shared depth between the two radar views) into transformer self-attention and cross-attention modules and outputs queries to support image-plane object detection and segmentation via calibrated or learnable radar- to-camera coordinate transformation and 3D-to-2D pinhole camera projection.
of 3D-embedding object
pose estimation [44]. It is noted that RF-Pose is not publicly accessible. More recently, RFMask [38] borrows the Faster R R-CNN framework [27] by proposing candidate regions only in the horizontal radar heatmap via a region proposal network (RPN). A corresponding proposal the vertical radar heatmap is automatically determined using fixed-height candidate region at the same depth the horizontal proposal. The combined horizontal and vertical proposals are then projected into the image plane for bounding box (BBox) estimation. In addition, RFMask calculates the BBox loss only over the 2D horizontal radar view and disregards features from the vertical radar heatmap for BBox estimation.
In this paper, we exploit features from both horizontal and vertical radar views for object estimation and segmentation and introduce Radar dEtection TRansformer (RETR) (Fig. 1). RETR extends the popular Detection Transformer (DETR) [3], which effectively eliminates the need for hand-crafted components such as mon-maximum suppression and proposal/anchor generation, the multi-view radar perception. More importantly, RETR incorporates carefully designed modifications to exploit the unique multi-view radar setting such shared depth dimension and the transformation between the radar and camera coordinate systems. Our contributions are summarized below:
Extending DETR for Multi- View Radar Perception: 1) Encoder: we associate features from both radar views by applying self-attention over the pooled multi-view radar features, eliminating the need for a cumbersome association scheme. We introduce a top- K feature selection allow only K features from each view keep the complexity low. 2) Decoder: the DETR decoder provides a natural way to associate the same object query to correspond- ing features from the two radar views via cross-attention. As such, the object query is able to learn 3D spatial embedding of objects in the radar coordinate (see Fig. 1).
2. Tunable Positional Encoding: To enhance feature association across the two radar views, we further exploit the fact that the two radar views share the depth dimension and introduce tunable positional encoding (TPE) attention map to prioritize the relative importance of depth dimension and avoid exhaustive correlations between radar views.
inductive bias. TPE imposes constraints in the
3. Tri-Plane Loss from Both 3D Radar Coordinate and 2D Image Plane:
we enforce the output queries of the DETR decoder to directly predict 3D BBoxes in the radar coordinate system and convert them into the 2D image plane. We introduce a tri-plane loss that combines the BBox loss in the 3D radar plane and that in the 2D image plane, to calculate the global set-prediction loss.
4. Learnable Radar-to-Camerr Coordinate Transformation: We employ calibrated radar- to-camera coordinate transformation via calibration process and a learnable coordinate transformation via reparameterization by preserving the orthonormal (i.e., 3D special or- thogonal group SO (3)) structure of the rotation matrix.
We demonstrate the effectiveness of our contributions through evaluations on two open datasets: the HIBER dataset [38] and the MMVR dataset [26].

---

2 Rader Views
3D Radar Coordinaze
3D Camera Coordinate
20 Image Plane
Dopith
Elvalion
Azemuthh
(6) eojectt
(6)
Aztmait
Depti
Radar to Camera Transforaatoo
Projecion p20
3D tD
Camera
Figure 2: Indoor radar perception pipeline: (a) multi-radar views are utilized to estimate 3D BBoxes in the radar coordinate system; (b) the 3D BBoxes are then transformed into the 3D camera coordinate system by a radar-to-camera transformation; and (c) the transformed 3D BBoxes are projected onto the image plane for final object detection. Blue line denotes ixed-heigh regional proposal in RFMask, while Magenta line denotes an object query with learnble height in RETR.
2 Related Work
Radar-based Object Detection and Segmentation: Indoor radar perception tasks include object detection (BBoxes), pose estimation (keypoints), and instance segmentation (human masks) [1, 35, 43, 44, 20, 23, 45, 38, 46], and radar datasets different data formats were reported in [35, 31, 30, 39, 40, 14, 38, 26]. Particularly, radar weatmap-based approaches have gained attention not only in indoor perception [43, 44, 14, 38, 26] but also for automotive radar perception [19, 24, 32, 11, due to richer semantic features compared to those extracted from sparse radar point clouds [31, 30, 39, 2, 15, 40, 41]. RF-Pose [43] predicts human poses on the image plane using a convolution autoencoder- based architecture. With the HIBER dataset [38], RFMask considers proposal-basee object detection and instance segmentation. More recently, MMVR [26] has been openly released to accelerate advancements in indoor radar perception.
Image-based Object Detection and Segmentation with DETR: Since the introduction of DETR for 2D image-plane object detection, subsequent studies have been developed based on its frame- work [21, 47, 4, 17, 37, 18, 10, 25], largely due to DETR's s ability to eliminate the need for hand- designed components such as non-maximum suppression (NMS). In [21], Conditional DETR de- composes the roles of content and positional embeddings in the transformer decoder, improving not only prediction accuracy but also training convergence speed. More recently, [25] has proposed Rank-DETR as rank-oriented architectural design, guaranteeing lower false positives and false negatives in prediction.
3 Preliminary
Generation of Radar Heatmaps: Conceptually, let us consider pair of (virtual) horizontal and vertical antenna arrays with Nan: elements for each array, sending set of frequency modulated continuous waveform (FMCW) pulses for object detection [26, 38, 34). The two 1D arrays generate one horizontal radar view in the azimuth-depth (x 2) domain and one vertical radar view in the elevation-depth (y 2) domain,
K M
Ky M
(m(aa)) A6
dm (v.e)
Yhor (t, x, 2)
)
Skmm
yver (t,
Skmm,
Am 1 m==1
Ar 1=
where 8k.m.t denotes the k-th sample of FMCW sweep on the m-th antenna at time I, A is the wavelength of the k-th sample, dm (x, denotes the round-trip distance from the m-th array element to position (I, z), and K, and M denote the number of samples and the number of array antennas, respectively. Usually, the azimuth
in an interval of E X' [xmin Ar Xmax] and the elevation and the depth z are similarly defined. At a particular time t, we have the horizontal radar heatmap Yoor(t) (t) Bhor (t, x, 1 RDDD and the vertical radar heatmap Y (t)
Lver (t, y, 2) |==Ez E R HD with a shared depth axis. The multi-view radar testbeds HIBER [38] and MMVR [26] utilize advanced MIMO-FMCW radar systems. We defer the MIMO-FMCW radar heatmap generation to Appendix D.
3

---

Horoow Siow
Top-K Selention
3D Boox
Tunable Fasitonal Fmibattiing
Transormss Encoder
Transfomer Decoder
Sespnentation readd
0 ODD
Top-s Selection
Ocjec:
Figure 3: The RETR architecture: 1) Encoder: Top-K features selection and tunable positional encoding to assist feature association across the two radar views; 2) Decoder: TPE is also used assist the association between object queries and multi-view radar features; 3) 3D BBox Head: Object queries are enforced to estimate 3D objects in the radar coordinate and projected to planes for supervision via a coordinate transformation; 4) Segmentation Head: The same queries are used to predict binary pixels within each predicted BBox in the image plane.
By taking I' consecutive multi-view radar heatmaps (Y R'' and Yeer E RTxxxx) as the input, we are interested in detecting objects on the image plane:
Indoor Radar Perception:
F Finaga projsmage (T (ʃ (Ynor, Yver)))
where Finage denotes predicted BBoxes for object detection and pixel-level masks for instance segmentation. Using the BBox as an example in Fig. 2, our pipeline includes the following steps: 1) Fig. 2 (a): By taking the two radar views over T consecutive frames (Y Yvez) input, the end-to-end object detection module f outputs a set of parameters describing 3D BBoxes in the radar coordinate system; 2) Fig. 2 (b): The radar-to-camera 3D coordinate transformation T converts the predicted 3D BBoxes at the output of f to corresponding 3D BBoxes in the 3D camera coordinate system; 3) Fig. 2 (c): The 3D-to-2D projection proj projects the 3D BBox in the camera coordinate system into corresponding 2D image plane normally with known pinhole camera model.
4 RETR: Radar Detection Transformer
We first present the RETR architecture and then highlight radar-oriented modifications. We defer the discussion Segmentation Appendix B.
4.1 RETR Architecture
present the RETR architecture Fig. 3, introducing major modules in left-to-right order. Refer to Appendix A for the detailed architecture.
Backbone: Given Yoor E RTxW'D and Yver E R shared backbone network (e.g.. ResNet [8]) generates separate horizontal-view and vertical-view radar feature maps: Zhor backbone (Yooz) E RCx x and Zver backbone (Yvex) E RCx # x where C and repre- sent the number of channels and downsampling ratio over the spatial dimension, respectively.
Feature Selection: A transformer-basee encoder expects sequence of features input. This done by mapping the feature maps into a sequence of P multi-view radar features H = (Hor, H } E CCP, ZBor - Haor RC' P.u. and Z Hver RCxP Peee, where P P Phor + Pver- We defer the discussion of top-K K feature selection to Section 4.2.
Encoder Cross- View Radar Feature Association: The transformer encoder provides simple yet effective method for associating radar features from both horizontal and vertical views by applying self-attention the pool of P multi-view radar features H (Hoo Hyer) E CxP eliminating the need for cumbersome association schemes. Specifically, the 1-th (1 = O.... Loelf - 1) encoder layer updates the multi view radar features through multi-head self-attention Att
H |+| H' + FFN H
H' .H Att (Que (H'). Key (H'). Val (H))), (3)

---

where FFN denotes feed-forward networks, Leeis is the number of encoder layers, and Que, Key and Val are projections derive the multi-head query, key and value embedding from H, respectively. For the first (0-th) layer, we have H = H. Note that we omit the description of "Layer norm" and "multi-head index" in Eq. 3 for clarity.
Additionally, since the multi-view radar features lack positional information and the self-attention is permutation-invariant,, we supplement H' with positional embedding added (or attached) to the input of each encoder layer. Refer to Section 4.3 for tunable positional encoding.
The decoder provides a natural way to associate the same object query with features from the two radar views via cross- attention. For each decoder layer, takes N object queries Q' {qı,... .qN RCXN as its input, and consists of a self-attention layer, a cross-attention layer and a FFN. Specifically for the (-th 0, 1,,..
Decoder to Associate Object Queries with Multi- View Radar Features:
Loross
1) decoder layer, it first updates all queries through multi-head self-attention
Q' ='
Q' Attsee (Que (Q') Key (Q') Val (Q')).
where Que, Key and Val are the projections with different parameterization from those in the self- attention layer (Eq. 3). Then, the decoder layer further updates the object queries Q' of Eq. via multi-head cross-attention with the multi-view radar features H Lonit from the encoder output:
= Q FFN (Q'). Q' = Q' Attcroaa (Que (Q') Key (H Val (HLL)) (5)
where both Q' and " are supplemented with positional embedding. Finally, the decoder outputs N enhanced object queries Q Qleine for downstream tasks.
Mapping from 3D Radar Coordinate to 2D Image Plane: Given the N enhanced object queries
Q Larons RETR directly estimates 3D BBoxes in the radar coordinate:
g {CD, cy, cz, w, h, d) sigmoid (FFN (q)). q Loua
(6)
where describes the 3D BBox center and respective widths along the 3D axes, and sigmoid normalizes the 3D BBox prediction to [0, 1]. Then, as shown in Fig. 2 (b), we apply a radar-to-cament transformation T to convert the predicted 3D BBoxes to ones in the 3D camera coordinate as
9 Jeaeera (z'anersa Weaners: )
) T
T (gresar) = Rgreder + t, i= 1,2, 8, (7)
where R is a 3D rotation matrix, e RR3 is the 3D translation vector, and giader is i-th corner of the 3D BBox corresponding g. Subsequently in Fig. 2 (c), we project the 3D BBoxes Iemeer onto the 2D image plane via a 3D-to-2D projection. From the projected 2D corners, one can calculate the 2D BBox center and width and height in the image plane as
Dinit {ex, cy, w. h3 T
proj image (Geaners)
(8)
The final BBox estimation Dinage in the image plane is obtained by adding an offset head FFN R.0 + R to compensate for the spatial downsampling and normalizing it to the interval [O, 1]:
b Dinage sigmoid (binst + FFN (binst ① g))
4.2 Top-K Feature Selection
In DETR, the sequentialization simply collapses the spatial dimensions of the feature map into a single dimension, resulting in Paor W D/s² and Pver HD/ s² features for the horizontal and vertical radar feature maps, respectively. As a result, we have P = (W + H) D/s multi-view radar features. It is known that the complexity of transformers grows quadratically with respect to the feature length P. Here, we introduce a customized Top- K feature selection, maintaining a low complexity for the RETR encoder and decoder: Hoor Selector (Zorz) RC and Hyer Selector (Zver) E RC*K where K & min WD/s 2 HD D/2)). In this case, we shrink the multi-view radar tokens from = (W~ + H)D/s² to P 2K. For each radar frame, we consistently select the Top-K strongest features, which may originate from varying spatial locations depending on the specific radar frame. Consequently, the gradient propagates back through the selected K features to the backbone weights, irrelevant to their spatial locations.
5

---

content
P: position
depth i A: angie)
D
+
02
D
A
D
Content Embedbing
Postional Embebding
Content
Paskional Embedding
A
A
08
Embeddiing ng
(a)
(b) Conditional
(c)
Figure 4: Schemes of positional encoding: (a) the operation in the original DETR; (b) the concatenation in Conditional DETR; and (c) TPE in RETR that allows for adjustable dimensions between depth and angular embeddings and promotes higher similarity scores for keys and queries with similar depth embeddings than those far apart in depth.
4.3 TPE: Tunable Positional Encoding
The TPE is built on the top the concatenation operation between the content embedding (cither feature embedding h the encoder or decoder embedding q at the decoder) and positional embedding (b)):
p in the conditional DETR [21] (see Fig.
(Cque 3 Pqua)
(Cxay ① Pkey) Cque Czay + P +Puue PquePkay:
(10)
where @ denotes concatenation, rather than the sum in DETR [3] (see Fig. 4 (a)):
T (Cque + Pquu) (Czey + Pkey) = Cque Czey + CquaPeey + Pque Czay + P T
Paua
(11)
It is that Eq. 10 eliminates the cross terms between the content and positional embeddings in Eq. 11 and, allowing content/positional embeddings focus on their respective attention weights, contributes faster training convergence [21].
In our case, the positional embedding is composed of a depth (y) axis and an angular (either azimuth elevation 2) axis. As such, d @ with d representing the depth positional embedding and a the angular positional embedding. Then expanding Eq. 10 with d @ a leads to
(quu θ dqua ① (qua)
(Czay ① daay ① (kayy
Cque Ckay + d dxay Quue Okay.
(12)
In Eq. 12, we have the following observations:
1. Eque Ckey reflects how similar the features in the key and query may appear;
2. Depth similarity duuedeey dhey remains consistent regardless of whether the key and query originate from the same radar view or different radar views;
3. Angular similarity a Okey can be self-angular similarity (azimuth-to-azimmlg or Devaaion-to-elevatin when the key and query are from the same radar view, cross- angular similarity (azieuth-to-ellevation or elevation-to-azimutti for different radar views.
Motivated by the above observations, we can promote higher similarity scores for keys and queries with similar depth embeddings than those far apart in depth, especially for the ones from different views, by allowing for adjustable dimensions between depth and angular embeddings:
daep = adgoe: dang =
az dpos
daep dang dgoe-
(13)
where the tunable dimension ratio a is in the interval [o. 1]. As illustrated in Fig. 4 (c), when a = 0.5. the positional embedding is equivalent to that used in conditional DETR. When a approaches the depth positional embedding is minimized, making the depth similarity di dzay negligible in Eq. 12. Conversely, a approaches 1, the depth positional embedding dimension increases, and does the importance of the depth similarity in Eq. 12.
We implement TPE with fixed sine/cosine positional encoding along the depth and angular (azimuth or elevation) dimension. For an even depth/angular positional dimension, we have
d3 = (paep// 22/dea))
d2 = cos(paeg/ 72 (l//
i = 0, 1, ... ddep/ 2 - 1,
(14)
a2 = sin(Pang/ / 7 2ildde)) 02i cos cos(Pang/) 2i/dmee)
i i 0. 1, ... dang / 1, (15)

---

where Paop/ang and ddap/ang are the position index and dimension for the depth and angular axes, respectively, is the (even/odd) element index, and 5 10000 is a temperature. By adjusting the ratio in Eq. 12, we change the dimensions of the depth d in Eq. 14 and the angular a in Eq. 15, while keeping the total positional dimension of p = d @ a constant. We show the visualization of TPE in Appendix C.
4.4 Tri-Plane Set-Prediction Loss
DETR calculates a matching cost matrix with each element constructed from classification cost Lelass and 2) a BBox loss between one N predictions b and one ground truth BBoxes b (including the "no object" class). The BBox loss is weighted combination of the generalized intersection over union (GloU) loss Laxeu [28] and the loss Lt,:
Loo b) = (otov b) + Lz, (b, b):
(16)
where X, denotes the weight. Over the permutation set Gx between N predictions and ground truth
objects, the Hungarian algorithm [12] is applied with the matching cost matrix find the optimal assignment o* G of predictions to ground truth. Given o² the loss is computed only for the matched pairs and is referred to as the set-prediction loss.
Since RETR predicts 3D BBoxes 9 in the 3D radar coordinate and maps them into the 2D image plane, we propose to enhance the above Hungarian match
Loss (Horizontal Radar Flane)
Proj
(Radar coordratee)
BBox bys coorrene buber ftane)
Flecatir
cost matrix using Tri-Plane BBox Loss from both the
radar coordinate and image plane. This is illustrated Fig. 5, where 3D BBox g in the radar coordinate is projected onto 1) the 2D horizontal radar plane as bnor = =proj (g) (the top branch); 2) the 2D vertical radar plane as bver = projvar (g) (the middle branch); and 3) the 2D image plane as Dimage of Eq. 9 (the bottom branch). The tri-plane BBox loss Lbs sums up 2D BBox losses over all three planes using Eq. 16:
Proj
Loos (Vertical Radar Plane)
Depth
nine B. (Vetical Radar Plane)
Aninwith
Trans
Relinemee
Loss (image
+ Proj.
BBo bun Orage
Final EBcx Prnse (rrage ('brn)
Plane)
Figure 5: Tri-Plane BBox loss.
Cori
Loox (bzor: bnox) + Loox (bvez, byer) Loox (binage: Binage)
(17)
RETR finds the optimal assignment oe using the matching cost with the original classification cost Lelaze and 2) the tri-plane BBox loss Larz. The resulting set-prediction loss using o is referred as the tri-plane set-prediction loss.
4.5 Learnable Eadar-to-Camera Coordinate Transformation
The rotation matrix R and translation vector t the radar-to citar-tt-camerz transformation of Eq. can be calibrated in advance. However, this calibration process may be accurate only for a limited interval of depth and angles. Instead of relying on the calibrated transformation, we introduce a learnable transformation via reparameterization
R while keeping it orthonormal. To this end, need to ensure that the learnalle R resides in the 3D special orthogonal group SO (3). Considering that SO (3) is special case of Lie group, one of the differentiable manifolds, we can firstly map 3D vector
{Waz, Wyy w E R to Lie algebra so (3) using the projection [-] R so (3).
And then we apply the exponential map exp so (3) + SO (3) that maps [w] into the nearest point in SO (3) such that the resulting exp ([w]) resides on SO (3) and satisfies the orthonormal
structure [13, 33]. This leads to the following reparameterizztioon of R. in terms w:
R. exp ((w) sin Φ
[w] +
cos Φ
(w)
s.t. (w) =
Wy
WW
Ww
(18)
d²
Wy
0
where o ||س|| is the & norm, With the above reparameterization, the learnable -o-mmer coordinate transformation in Eq. 7 reduces to learn the vector and the translation vector t.

---

Table 1: Main results of object detection in the image plane under "P2S1" MMVR. The top section shows results from conventional models, while the bottom section presents RETR results.
Model
Dim
Input
BBox Loss
AP
AP 50
AP
AR;
AR30
RFMask
2D
H,
H+1 I
31.37
61.50
27.48
33.23
38,41
DETR
2D
H
H+I
29.38
62.31
25.35
31.32
43.06
DETR (Top-K)
2D
H, V
H+I
39.71
82.74
33.29
38.98
52.81
RETR (TPE@Ded Dec.)
3D
H, V
B++++
45.94
81.99
44.04
42.03
57.38
RETR
3D
H. V
H+V+I
46.75
83.80
46.06
42.19
57.39
Table 2: Main results of object detection in the image plane under "WALK" of HIBER. The notation
follows the same format as Table 1.
Model
Dim
Input
BBox Loss
AP
APso
AP75
AR:
AR10
RFMask DETR
2D
H.
H+I
17.77
52.46
6.78
32.71
32.71
2D
H
H H+I
14.45
47.33
4.25
28.64
28.64
DETR (Top-K)
2D
H, V
H+I
14.35
48.94
5.50
28.78
28.78
RETR (TPE@Dec)
3D
H. V
H V
20.18
52.53
7.32
32.91
32.91
RETR
3D
H. V
22.09
59.83
10.99
35.16
35.16
5 Experiments
5.1 Setup
Datasets: We evaluate performance over two open indoor radar perception datasets: MMVR [26] and HIBER [38]. MMVR includes multi-view radar heatmaps collected from over 20 human subjects across 6 rooms over span of 9 days. In our implementation, we utilize data from Protocol (P2) which includes 237.9K data frames capturing both single and multiple human subjects diverse activities such walking, sitting, stretching, and writing on the board. For the training-valicn vaniing-vlidattion-test split, we follow the data split S1 as defined in MMVR.
HIBER, partially released, includes multi-view radar heatmaps from 10 human subjects in a single room but from different angles with two data splits: 1) "WALK", consisting of 73.5K data frames with one subject (Section 5.2); and 2) "MULTI", consisting of 70.8K radar frames with multiple (2) human subjects walking in the room (Appendix G). More dataset details can be found in Appendix
Implementation: We consider RFMask [38] and DETR [3] as baseline methods. Since RFMask
and DETR originally compute the BBox loss only in the 2D horizontal (H) radar plane and the 2D image (I) plane, respectively, we enhance both methods with unified bi-plane BBox loss (H + I). We also introduce DETR variant with top-K feature selection, allowing
take features from both horizontal (H) and vertical (V) heatmaps input. For RETR, we set K 256 for the top-K selection, the positional embedding dimension dpes 256, and a tunable dimension ratio include one variant that only employs the TPE at the decoder (TPE@ Dec.). More settings can be found in Appendix E.
0.6. We
For object detection, we adopt average precision (AP) at two IoU thresholds of 0.5 (APso) and 0.75 (AP7s) and its mean (AP) over thresholds [0.5 0.05 0.95). We also consider average recall (AR) when it is restricted to making only one detection (AR,) or to 10 detections (AR4o) per image. For segmentation, we report the average IoU value between the predictive and ground truth masks. Detailed metric definitions can be found in Appendix F.
Metrics:
5.2 Main Results
MMVR: Table shows the main results the MMVR dataset under "P2S1". Compared with RFMask, DETR with a single horizontal radar view does not show performance improvement. By
httep:://eaadd.oor///eoo///////
http:s//ggthabb.mmm/tttlillggge

---

Query:
Query: 2
Hor. Plane
Predicted BBoxes on Image Plane
Ver. Plane
Query:
Query: 2
Query
Depth
Depth
Aziutth
Hlevation
Figure 6: Visualization of cross-attention map between predicted BBoxes and multi-view radar features. BBoxes with the same color correspond to the same subject.
Ratio
0.2
0.4
0.6
0.8
LT
AP
AR;
Loss
AP
AR1
AP
44.35
45.16
46.75
43.81
42.97
40.20
H+I I
42.18
39.39
AR1
42.04
41.60
42.19
41.89
46.75
42.19
H+V+I
46.75
42.19
(a) A dimension ratio a achieves the best.
0.6 of TPE (b) Learnable Transforma- (c) Tri-plane loss enhances ob- tion (LT) may replace calibra- ject detection. tion.
Table Ablation studies under "P2S1" on MMVR.
just adding the vertical radar view at the input, DETR with top- K selection exhibits a noticeable performance improvement over RFMask. Built upon DETR (Top-K), RETR (TPE Dec.) imple- ments two enhancements: 1) TPE at the decoder and 2) tri tri-plane BBox loss, resulting in further improvements with gain of 6.23 in AP, 10.75 in AP7s, and 4.57 in AB10. highlighting the importance of TPE and supervision at the vertical radar view. By further incorporating TPE at the encoder, the full version of RETR achieves impressive performance improvement RFMask, demonstrating increases of 15.38 in AP, 22.30 in AP50. and 18.58 in AP75, respectively. The results under "P2S2" on MMVR can be seen in Appendix G.
Table presents the main results the HIBER dataset under "WALK". Similar to Table 1, we observe a similar trend of performance improvement from DETR to RETR variants. Numerically, see increases of 4.32 in AP, 7.37 in APsa. and 4.21 in AP when directly comparing RETR to RFMask. These performance improvements are smaller compared with those in Table 1. This is potentially because the HIBER data under "WALK" predominantly involves walking, where RFMask's s fixed-height vertical proposals may work fine. In contrast, MMVR under "P2" includes more diverse activities such as sitting, leading to likely overestimated vertical proposals for RFMask and thus greater improvements in MMVR than HIBER. The results under "MULTI" on HIBER can be seen in Appendix G.
HIBER:
Visualization of layer between predicted BBoxes (via object queries) and multi-view radar features. RETR accurately predicts the subject in the background of the image plane (middle panel) with forward-bending posture (Query 1). The cross-attention maps of Query 1, with respect to horizontal (left) and vertical (right) radar features, highlight areas with features contributing the most to Query These contributing areas in the vertical plane are more stretched along the depth axis compared with those in the horizontal plane. Notably, the contributing areas from the two views share similar depth intervals. For Query 2 which identifies the subject in the foreground, the cross-attention maps shift its focus to contributing areas at closer depth compared with those for Query I, indicating an effective 3D spatial embedding of object queries at the RETR output. We provide more visualizations in Appendix H.
Map: Fig. 6 presents the cross-attention map the last decoder
We present failure cases in Fig. 15 of Appendix H. Predicting arm positions remains challenging, suggesting that RETR may not focus its attention on regions with weak radar reflections. Moreover, multi-path reflections from the ground, ceiling, and other strong scatterers (e.g., metal) can cause (first-order or second-order) ghost targets and elevate the noise floor. Traditional signal processing techniques can mitigate these effects but require access to raw radar data. Alternatively,
Limitation:

---

ghost targets can be labeled in the multi-view radar heatmaps, though this can be time-consuming and costly. One can then extend RETR classify output queries to one of { person, ghost), alongside regressing queries to the BBox parameters.
5.3 Ablation Studies
We report ablation studies with RETR under "P2S1" on MMVR. Further results of ablation studies can be seen in Appendix G.
Tunable Dimension Ratio a: Table 3a presents the ablation study of the tunable dimension ratio a and its impact on the object detection performance in terms of APso ( prriarry vertical axis) and AP and APrs (secondary vertical axis). The results indicate that 0.6 yields the best performance. The detection performance gradually decreases as approaches to 0 and 1.
Learnable Transformation (LT): To evaluate the effectiveness of the Learnable Transformation in Section 4.5, we compare AP and AR; metrics of RETRs with and without LT. The results in Table 3b indicate that it is possible to incorporate the radar-to-camera geometry into the end-to-end radar perception pipeline without the need for cumbersome calibration step, while still achieving comparable perception performance.
Table 3c compares RETR with a bi-plane BBox loss (horizontal radar plane and image plane) to that with the tri-plane loss (including the vertical radar plane). The results highlight the necessity of accounting for the vertical BBox loss and the importance leveraging features from the vertical radar heatmap, leading to a performance improvement of 4.47 AP.
Tri-Plane Loss for RETR:
6 Conclusion
In this paper, we introduced RETR, extending DETR to the multi-view radar perception with carefully designed modifications such as radar and camera coordinates, and a learnable radar-to-camera transformation. Experimental results over two radar datasets and comprehensive ablation studies demonstrate that RETR significantly outperforms both RFMask and DETR baseline methods.
feature similarity via TPE, tri-plane loss from
Broader Impacts: Indoor radar perception technologies, including RETR, offer a wide range of social applications in navigating and monitoring subjects such as the elderly, infants, robots, and humanoids, enhancing safety and energy efficiency while preserving privacy. However, it is crucial that perception results remain secure and private to prevent misuse in inferring subject attributes such as gender, size, and height. These technologies could potentially be used to advance indoor surveillance without individuals' acknowledgment consent.
References
[1] Fadel Adib, Chen- Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand, Capturing the human figure through wall. ACM Trans. Graph., 34(6), 2015. URL https://doi. org/10..0. 1145/ 2816795. 2818072.
[2] Sizhe An. Yin Li, and Umit Ogras. mRI: Multi-modal 3D human pose esti- mation dataset using mmWave, RGB-D, and inertial sensors. In Advances in Neural Information Processing Systems, volume 35, pp. 27414-27426. 2022. URL
httpss//prooeeediggs
www.eeiinnnnnnnnns...ee.e neurips. cwterips.ccppaapee www.cc/ppperiiiieeeeeee fiwesfllepppele Paperr aatp and mard__Beeeharss_e
pdf.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Confer- ence on Computer Vision (ECCV), pp. 213-229, 2020. URL :/dotss://oi..... org/ 2erg/10.10 1007/
100-3--330-8455---5
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF
10

---

International Conference on Computer Vision (ICCV), pp. 9630-96400 2021. URL https:
//doi.org////// 10. (..09//CCV489222 20211 00951.
[5] Fangqiang Ding, Xiangyu Wen, Yunzhou Zhu, Yiming Li, and Chris Xiaoxuan Lu. RadarOce: Robust 3D occupancy prediction with 4D imaging radar. arXiv:2405.14014, 2024. URL attps://arxie org/
14014.
[6] Mark Everingham, Lue Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisser- man. The PASCAL Visual Object Classes (VOC) Challenge. International Journal of Computer Vision, 88(2):303-338, 2010. URL https: //doi. http///ii.ooooo/////o 10. s0000//s11333.1. 000-00000000000 0275-4275-44
[7] Sevgi Zubeyde Gurbuz and Moeness G. Amin. Radar-based human-motion recognition with deep learning: Promising applications for indoor monitoring. IEEE Signal Processing Magazine, 36(4):16-28, 2019. URL https://ddo org/10 10. 1ww.111999MPPPP2 201.///PP......... 2890128.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. URL httph: owgooo///// 10. 1109/CVVP, 2016.2006.0.. 90.
(9] Jan Hosang, Rodrigo Benenson, Piotr Dollár, and Bernt Schiele. What makes for effective detection proposals? IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(4): 814-830, 2016.
10. 1109/11TTTT APPM.. 2015. 2465908.
[10] Zhengdong Hu, Yifan Sun, Jingdong Wang, and Yi Yang. DAC-DETR: Divide the attention layers and conquer. In Advances in Neural Information Processing Systems, volume 36, pp. 75189-75200, 2023. htRL:ttts:////oceeeeeeee ( Panno--aaaa----eeeeeeeeeeee pdf.
neur ips cc/ pec/paper_filee papeerr//////aaaa
[11] Seung-Hyun Kong, Dong-Hee Paek, and Sangyeong Lee. RTNH+: Enhanced 4D radar object detection network using two-level preprocessing and vertical encoding. IEEE Transactions on Intelligent Vehicles, pp. 1-14, 2024. URL https: //10. 1109/TIV .024..22.0 3428696.
[12] Harold W. Kuhn. The Hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83 97, 1955. URL httpst::: //to:::::///..ooo http//o...o.ogg//// org/10.../000 10w.0/0009997778 0tt./9978----40--677---0-0
[13] John. M. Lee. Introduction to Smooth Manifolds. Springer, 2003, URL https://doi.orr 10. 0911/99881-4499 9982-5.
[14] Shih-Po Lee, Niraj Prakash Kini, Wen-Hsiao Peng, Ching Wen Ma, and Jenq-Neng Hwang. HuPR: A benchmark for human pose estimation using millimeter wave radar. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5715-5724, 2023. URL org/10 10. /WW.///6CVC/.../// 2023.0222222 00567.
[15] Peizhao Li, Pu Wang, Karl Berntorp, and Hongfu Liu. Exploiting temporal relations on radar perception for autonomous driving. In JEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. pp. 17050-17059, 2022. URL https: // /pwp//ccppcccccccepeeppe/////ase thecvf. com/ com/content wwtteeet/////222// http://////pppeerssii..p.eeooiie wtw.ttg_geeaaalllaeee
Betaalaeeeeaaatteaealeeee _n__arrr_________ wttpr//nwwo......iii...o........ Ltt.C_/___2222_pe_e__e
1002__papr..dff.. pddf.
[16] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2999-3007, 2017. URL htttp:://dae org/10..111.... 1109/ ICC.///CC.... .017. .24.
[17] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for DETR. In International Conference on Learning Representations (ICLR), 2022. URL http:::/ net/ wwt.:/rummmmmmeme// form?id= oMI911
[18) Shilong Liu, Tianhe Ren, Jiayu Chen, Zhaoyang Zeng, Hao Zhang, Feng Li, Hongyang Li, Jun Huang, Hang Su, Jun Zhu, and Lei Zhang. Detection transformer with stable matching. In IEFF/CVF International Conference on Computer Vision (ICCV), pp. 6677, 2023. URL https:/ /ttps://doi.. org/10. 11109//CC550000.2223...... 00597.

---

[19] Yang Liu, Feng Wang, Naiyan Wang, and Zhao-Xiang Zhang. Echoes beyond points: Unleashing the power of raw radar data in modality fusion. In Advances in Neural Information Processing Systems, volume 36, pp. 33964-53982
2023. URL. httpp:://proceeddngggge// http://prroeedddn..u.ie.pppppppp neurips cc/ cecpppperfiless_s__ wtt.f//eppeee///////////// pppppppp2//////eeee a8f7f wtw.//99000227766666626262-----. Paper-Conference pdf.
[20] Chris Xiaoxuan Lu, Muhamad Risqi U. Saputra, Peijun Zhao, Yasin Almalioglu, Pedro P. B. de Gusmao, Changhao Chen, Ke Sun, Niki Trigoni, and Andrew Markham. milliEgo: single- chip mmwave radar aided egomotion estimation via deep sensor fusion. In The 18th Conference on Embedded Networked Sensor Systems (SenSys), pp. 109 122, 2020. URL https: //doi. org/10. 4444/3333419 3430776.
[21] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional DETR for fast training convergence. In IEEE/CVE International Conference on Computer Vision (ICCV), pp. 3631-3640, 2021. URL https://doii org/10. 1109/ICCV4882 2021.00363.
[22] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In International Conference on 3D Vision (3DV), pp. 565-571, 2016. URL https: 1109/3DV 2tt///3V..01...... 2016.79.
[23] Arthur Ouaknine, Alasdair Newson, Patrick Pérez, Florence Tupin, and Julien Rebut. Multi- view radar semantic segmentation. IEEF/CVF International Conference on Computer Vi- sion (ICCV), pp. 55611-16600, 2021. URL. https:/// org/ 10. 01538.
2021.
[24] Dong-Hee Pack. Seung-Hyun Kong, and Kevin Tirta Wijaya. K-Radar: 4D radar object detection for autonomous driving in various weather conditions. In Advances in Neural Information Processing Systems, volume 35, pp. 1819-3829 2022. URL https://proceeeee 185fdf (555dd22777a222bb33226661111111
www.iin....pp... cc/ pappaper__a. httpp/pppprreeeeeee wol.fflllppppe wwwes//apee/////// https/apper2222//111/ Paper-aataaaee
and and___nd_________ Benchmarkshsseeeeeee pdf.
[25] Yifan Pu, Weicong Liang, Yiduo Hao, Yuhui Yuan, Yukang Yang, Chao Zhang, Han Hu, and Gao Huang. Rank-DETR for high quality object detection. In Advances in Neural Information Processing Systems, volume 36, pp. 16100-16113, 2023. URL https: https://proceeddngs neurips http:/aprrfffee//////ppapeeeeeee fil.fiils/pppee wills//aapr//22//Eiii. 011774799621888888688880.
Paper- coree-Coferrre conference.p. .df.
[26] M. Mahbubur Rahman, Ryoma Yataka, Sorachi Kato, Pu Wang, Peizhao Li, Adriano Cardace, and Petros Boufounos. MMVR: Millimeter-wave multi-view radar dataset and benchmark for indoor perception. In European Conference on Computer Vision (ECCV), pp. 1000-22,,222 2025. ISBN 978-3-031-72986-7 URL hattps://dol http:/::/ddi..o///////// -01.79997----001-7778------1
[27] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 137-)1137- 2017. URL
org /t.g////...100 TPAMI.
2016. 2577031.
[28] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and loss for bounding box regression. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 658-666,
2019. URL https: httpps://ddo.orrrt org/ 10. 1109/CVPR.... 2019. .00055.
[29] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional net- works for biomedical image segmentation. In Medical Image Computing and Computer- Assisted Intervention (MICCAI), pp. 234-241, 2015. URL https: htte::/ddoi.rrt orw.i..rg1..... 10. 1007/
978-3-319- 24574-4_28.
[30] Arindam Sengupta, Feng Jin, Renyuan Zhang, and Siyang Cao. mm-Pose: Real-time human skeletal posture estimation using mm Wave radars and CNNs. IEEE Sensors Journal, 20(17): 10032-10044, 2020. URL /ttop: org/10 1109/ JSEN 2020. 2991741.
12

---

[31] Akash Deep Singh, Sandeep Singh Sandha, Luis Garcia, and Mani Srivastava. RadHAR: Human activity recognition from point clouds generated through millimeter-wave radar. In The 3rd ACM Workshop on Millimeter- Wave Networks and Sensing Systems, mmNets '19, pp. 51-56 2019. URL wttthh::///oooooo wrg/.o.g//o.......o /0. 1145/ 444452334624..... 3356768.
[32] Mikael Skog, Oleksandr Kotlyar, Vladimír Kubelka, and Martin Magnusson. Human detection from 4D radar data low-visibility field conditions. arXiv:2404.05300 2024. URL https:
/ //ar //axiiv... berg/abs//i 2404. 05307.
[33] Joan Solá, Jeremie Deray, and Dinesh Atchuthan. A micro Lie theory for state estimation in robotics. arXivv 11X2.0///22 2021. URL https::/aaaaat //axivv.//iiivvv/va org/ aas///////// ttbs/188 2/12201537,
[34] Shunqiao Sun, Athina P. Petropulu, and H. Vincent Poor. MIMO radar for advanced driver- assistance systems and autonomous driving: Advantages and challenges. IEEE Signal Process- ing Magazine, 37(4):98-)))))) 117, 2020. URL https:: //10. 1109/MSP. 2020 2978507.
[35] Baptist Vandersmissen, Nicolas Knudde, Azarakhsh Jalalvand, Ivo Couckuyt, André Bourdoux, Wesley De Neve, and Tom Dhaene. Indoor person identification using a low-power FMCW radar. IEEE Transactions on Geoscience and Remote Sensing, 56(7):3941-399) 2018. URL
https://doi. org/10. 1ww.//199TTTRR 2018. 2816812.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30, 2017. URI. https: p neurips cc/ Paper .dff
paper files/ wtt.e//ppppppeeoeeeeeeeeeeeeeeee
[37] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor DETR: Query design for transformer-base detector. AAAI Conference on Artificial Intelligence, 36(3):2567-257,, 2022.
URL http:: //doi. org/ 10. 1609/ /aali./aaa.... v3613. 20158.
[38] Zhi Wu, Dongheng Zhang, Chunyang Xie, Cong Yu, Jinbo Chen, Yang Hu, and Yan Chen. RFMask: A simple baseline for human silhouette segmentation with radio signals. IEEE Transactions on Multimedia, 25:4730 2023. URL httpss////// http://dooooo/////// org/10.../.. 1109/TMM.
[39] Hongfei Xue, Yan Ju. Chenglin Miao, Yijiang Wang, Shiyang Wang, Aidong Zhang. and Lu Su. mmMesh: towards 3D real-time dynamic human mesh construction using millimeter-wave In org/rog./// 10. 1041111443444 /455//455864... 3467679.
MobiSys, pp. 269-282, 2021. URL https http://ttooo//.....//....
[40] Jianfei Yang, He Huang. Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Sheng- hai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie. MM-Fi: Multi- modal non-intrusive 4D human dataset for versatile wireless sensing. In Ad- vances in Neural Information Processing Systems, volume 36, pp. 18756-18768, 2023. URL https: https:///proceedings wwwpeeeddiiniinnninnneeeeeeeeeee neurips. www.ips..ccpppppr... fil.eillleeeeeee wwt.lfffleeeeeeeeeeeeee (at.99/00000AA1ee284444222222200 Pappeeee-----P-r---e-eeeee--eeee Dww--aattatttaeeeeeeeeeeeeeeeeee and haw.eeetsseaddeeeeeeaaeeeeeeee pdf.
[41] Ryoma Yataka, Pu Wang, Petros Boufounos, and Ryuhei Takahashi. SIRA: Scalable inter-frame relation and association for radar perception. In /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15024-15034, 2024. doi: 10.1109CCVPR32233..24000
URL https://openaccess thecvf wwt. http://cnneee//////////////eeeee http://////aaperraaaaaaaa_a Intaabl-anntee--rraeee aalaaaiionnan_ssscciaiiiiii.oo.a www.dsssssoctiosssaaooaaeoeeee for Radar Perception_ WVPR_2024_papeee
[42] Mingmin Zhao, Shichao Yue, Dina Katabi, Tommi Jaakkola, and Matt T. Bianchi. Learn- ing sleep stages from radio signals: A conditional adversarial architecture. In Interna- tional Conference on Machine Learning (ICML), volume 70, pp. 4100-4109, 2017. URL mlr ppwss/v70/2hao/ html.
attps://proceedin
[43] Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. Through-wall human pose estimation using radio signals. In IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7356-7365, 2018. URL https: //to:://ooiiieoo http://tto.oo....//// 10w.//119//// 1119/CVPR9../// 00768.
13

---

[44] Mingmin Zhao, Yonglong Tian, Hang Zhao, Mohammad Abu Alsheikh, Tianhong Li, Rumen Hristov, Zachary Kabelac, Dina Katabi, and Antonio Torralba RF-based 3D skeletons. In The 2018 Conference of the ACM Special Interest Group on Data Communication (SIGCOMM), pp- 267-281, 2018, https//t::/ htap:p:://oo... org/ 10. 1145/3230543 3230579.
[45] Peijun Zhao, Chris Xiaoxuan Lu, Jianan Wang, Changhao Chen, Wei Wang. Niki Trigoni, and Andrew Markham. Human tracking and identification through millimeter wave radar. Ad Hoc Networks, 1 11,, 2021. URL https: / /doi. org/10. 1016/1 adhoc. 2021. 102475.
[46] Peijun Zhao, Chris Xiaoxuan Lu, Bing Wang, Niki Trigoni, and Andrew Markham. Cubelearn: End-to-end learning for human motion recognition from mm Wave radar signals. IEEE Internet of Things Journal, 10336):103 2023. URL www.h:ttt//oooo org/10.1109/
JIOT, 2023. 3237494.
[47] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations (ICLR), 2021. URL https: http:////////ee//////t//// net/ forum?ids g29hCDWe6ke.
14

---

A Details of RETR Architecture
Transformer Encoder and Decoder: Fig. illustrates the transformer encoder and decoder used in RETR. In the original DETR implementation, the image features from the CNN backbone are given in input the transformer encoder, with spatial positional embeddings added to the queries and keys at each multi-head self-attention layer of the encoder. On the other hand, RETR extracts features from shared-weight backbone for both horizontal and vertical views and obtains them as [P..... PW WDsa, PT. P/D//>1 J. At this time, the positional encoding (TPE) is concatenated with the features (content). Subsequently, Top-K selection is applied to extract the most relevant features and reduce time and space complexity (i.e., Ph P Pg and P% in the left figure). These Top-K K features from the horizontal and vertical views concatenated compose single sequence of tokens, which are then fed the transformer encoder. The encoder consists a stack of multi-head self-attention layers, that allow for the consideration of correlations between the two views. The multi-head attention is simply the concatenation of M single attention heads followed by a projection layer L to regain the initial dimensionality. The common practice [36] is to use residual connections, dropout, and layer normalization:
mhAtt =layernorm Que (H) + dropout (LH))
(19)
H =Att (Que (H), Key (H), Val (H), W $ ...
+ Att (Que (H), Key (H) Val (H), WM),
(20)
where @ concatenation along the channel axis, and W denotes the weight tensor of attention.
The decoder receives the decoder embeddings, which we initially set to zero and concatenated with the object queries, and encoder memory (i.e. the output sequence of the encoder transformer), generating refined embeddings through multiple multi-head self-attention and cross-attention layers. In particular, the cross-attention layer utilizes the encoder memory to produce Keys and Values, which correlate with the Queries to produce the Refined Queries. In right figure of Fig. 7, the decoder embeddings which concatenated with the object queries are first input into the self-attention, and the output is then passed through a normalization layer. At this point, the values are added using a residual structure. Next, cross-attention between the encoder memory, used as the key, and the decoder embeddings is calculated. Similarly, residual structure is employed as in the self-attention. This entire sequence repeated Teroee times obtaín the final decoder embeddings.
Computational Complexity Following the computational complexity notation used in the DETR paper, every self-attention mechanism in the encoder has complexity of Odd2K d(2K))) where
Encoder
Decoder
Refined Queries
Ado Norm
Adid Norm
Sell-Attention
Q
Arcss-Attetion
X
K
x
Too K Selection
Too-- Selection
Self seff-Attentooo
TPC
K
Shared
Object Queries
Hortzontal View
Velt View
Non-Leamabl Modure
Learnalle Module
Figure 7: Illustration of (left) encoder and (right) decoder of RETR.
15

---

Segmentation Head
Res5
Res/
Res
Res.2
Cross-Abenticn
Reconstruce ructuctiin with U-Net
FPN CNN
Decoder
Transformation Projection
Embeddiing
8Box in Radar Plane
FFN
Class Score
Fixed Param Module
Learnao e Module
Figure 8: Illustration of segmentation head.
is the embedding dimension and is the number of selected features from the Top- selection. The cost of computing single query/key/value embedding is O () (with d Md where M denotes the number of attention heads and d' the dimension in each head), while the cost of computing the attention weights for one head is (d' (dd'(2K)²) Other computations may be negligible. In the decoder, each self-attention mechanism has a complexity of O + dd²) where N is the number of queries, and the aross-attention between query and multi-view radar features has complexity of ( (N 2K) + K). KKK.K. In conclusion, the overall complexity of our RETR model is
O4dd 4dK² 2d N dN2 + 2dNK).
(21)
B Segmentation
Architecture of Segmentation Head: The original DETR is naturally extended by adding a segmentation head on top of the decoder outputs. Following this extension, RETR enables segmentation by adding an architecture with similar structure. Fig. 8 illustrates the segmentation architecture implemented, consisting of cross-ates cosss-attetionn layer, feature pyramid network (FPN)- style CNN, and final light U-Net [29]. Given a single refined query, we use a cross-attention layer low resolution. For the backbone output used in
generate attention heatmaps for each object cross-attention, we utilized features extracted from the vertical heatmap, enhancing robustness to the height of the human. To increase the resolution of the mask, an FPN-style architecture is employed which also exploits the low-level backbone features at different layers (from to 2) to generate some coarse segmentation masks. Since the FPN module is also responsible for lifting features from the radar view to the image plane, it does not have enough capacity to generate fine-grained segmentation masks. Thereby,
also add very light U-Net to further refine the previously generated masks. is important to note that our model, differently from the original DETR implementation, predicts single binary mask for each query. Indeed, we exploit for each query the corresponding bounding box prediction in the radar plane, apply the Radar-to-Camera transfornation and the 3D-to-2D image projection, to obtain bounding box the corresponding portion from the ground truth segmentation mask, which is employed to supervise the segmentation prediction for the same query. As a loss function, we adopt the DICE/F-1 loss [22] and focal loss [16].
image plane. This bounding box finally used to extract
Training: We note that the segmentation head can be trained at the same time as the BBox head in an end-to-end manner, or we can first train the detection head and then freeze weights and train only the segmentation head in two-step process. We followed the original DETR and employed
16

---

Table 4: Segmentation results under "P2S1" on MMVR.
Model
Dim
Input
BBox Loss
IoU
RFMask
2D
H, V
H + I
65.30
DETR
2D
H
H + I
70.15
DETR (Top-K)
2D
H,
I
75.76
RETR (TPE@Dec)
3D
H.
H
76.16
RETR
3D
H.
+++++
77.21
RGB
GT
RETR
d6s3/006
48s6/002
d7s5/006
Figure 9; Visualization of segmentation results.
the latter strategy. During prediction, we filter out the detection with confidence below 50% then compute the per-pixel argmax to produce the final binary segmentation mask.
We report quantitative results the segmentation tasks in Table 4. From this table, RETR (which combines all our contributions) achieves 77.07 @ IoU, which is a significant perfor- mance improvement over the conventional RFmask with gap of 11.77@ IOU. In addition, we point out how the DETR (Top-K)) version (row 3) alone is able to increase the performance by almost 5%. We visualize the segmentation results in Fig. Each row represents the data segment number in MMVR. It can be observed that RETR captures the shape of people with high fidelity. Notably, the results for d6s3 and d8s6 demonstrate that we are able to segment even complex postures, including sitting positions. Additionally, shown in d7s5, RETR accurately estimates positions even when subjects are sitting far from the radar, such as the back of the room. These results indicate that RETR can be easily extended from detector to segmentation model by adding segmentation head, and it can accurately estimate masks. For more visualizations, including comparison with RFMask and failure cases, Appdendix H.
Main Results:
17

---

Rostfoon Frineddings
0.2
0.5
0.8
1.0
Ang
Dep
Ang
Des
Ang.
Dep
Ang.
Dep.
Ulimnssio dgsa
Similaa Matrix
Postion
Ang. Angle Fosiioona inbeekiing Dep. Desth Poskional Ombecding
Figure 10: Visualization of TPE: (top row) positional embeddings for each axis; depth and angle, (bottom row) similarity through positions with dot product of positional embeddings. Each column denotes the a (a = 0.0, 0.2, 0.5, 0.8, 1.0). The blue color represents the large value, and red color represents the low value. The range is 1-,
C Visualization of TPE
We visualize the positional embedding of each axis to observe the TPE. calculated the positional embedding according Eq. 14 and Eq. 15, and visualized each axis separate figure with several value of a (a = 0.0, 0.2, 0.5, 0.8, 1.0). Fig. 10 shows the results. The top row positional embeddings for each axis; depth and angle, and the bottom row is similarity through positions with dot product positional embeddings. Each column denotes the a (a = 0.0, 0.2, 0.5, 0.8, 1.0). The blue color represents the large value, and red color represents the low value. The top show that the characteristic elements are concentrated in the first some dimensions from top row. In addition,
0.8 and 1.0 expected contain more depth features since the spread of depth features larger than a = 0.5 or lower. Furthermore, when we look at the similarity matrix (bottom row), the deeper blue color is concentrated in the center of the matrix indicates that the depths are more closely matched to each other, and that the degree of similarity can be changed by changing the
0.8 and
1.0. This
D Multi-View MIMO-FMCW Radar Heatmap Generation
Fig. 1l illustrates the preprocessing flow of the multi-view radar heatmap using data from two MIMO-FMCW radars, which create two orthogonal virtual arrays composed of 86 ele- ments spaced at waveenaae in- tervals while transmitting multiple pulses. By sampling the pulses re- flected back, a 3D data cube can be formed, which is structured along the horizontal/vertical arrays, ADC sam- ples (intra- iinrr--ulse or fast-time), and pulse samples (inter-pulse or slow-time). Performing a 3D fast Fourier transform (FFT) on this data cube yields radar spectra across the angle (azimuth for horizontal radar and elevation for vertical radar), range, and Doppler velocity domains. The SNR is further improved by integrating radar spectra along the Doppler domain, resulting in two radar heatmaps azinuu and range- elevation) in polar radar coordinates. These heatmaps are then projected into the radar Cartesian coordinate system.
FMOW puise
Pulsos
Doppler
Horizontal Heatrea Ve geuaeen
ADC samolés
Depth
Depii
Hortorntll
Azemuth
Horizontal array
Sanpiig
sray
Azimuth
FMCW pulse
ADC sarplees
Pulser
Deoplet
RouR Reue (ENUAA auEREnnnee
Depth
Depih
Elevation
Bleatton
Figure 11: Multi-view heatmap preprocessing.
18

---

E
Details of Experimental Settings
MMVR Dataset: MMVR [26] has 345K data frames collected from 25 human subjects over 6 different rooms (e.g, open/cluttered offices and meeting rooms) spanning over 9 separate days. MMVR consists of 2 parts: 1) 107.9K data frames of protocol (P1): Open Foreground in a single open-foreground space with single subject; and 2) 237.9K data frames of protocol 2 (P2): Cluttered space in 5 cluttered rooms with multiple subjects and multiple actions, including sitting postures. Data splits are set same as S1 in MMVR. "PI" is used to establish the best possible radar perception benchmarks, while "P2" is designed for more challenging scenarios and for gross-environman and cross-subject generalization. The "P2" includes data such sitting postures; therefore, select this as the main dataset that we use in our experiments.
HIBER Dataset: HIBER [38] an open-source multi-view radar dataset including horizontal and vertical radar heatmaps and annotations such as 2D and 3D poses, BBoxes, and segmentation masks. Among its data splits, "WALK" and "MULTT"" are currently accessible. The "WALK" split includes 73.5K data frames, each featuring a single person per frame, while "MULTT" consistently includes two individuals per frame. We refined the original BBox labels in the HIBER dataset, addressing their initial overestimation by creating tighter BBoxes; see Fig. 12 for an illustration.
Figure 12: Original (left) versus Refined (right) BBoxes in the HIBER dataset.
The hyper-parameters used in our experiments of Section are shown in Table 5. The table is divided into three parts, Data, Model, and Training, each with parameter names, notations, and values for each dataset.
Hyper-parameters:
RFMask with Refined BBoxes: We use RFMask [38] conventional method for BBox and segmentation tasks. However, RFMask can only predict relaxed BBoxes in the image plane due to its loss calculation being limited to the horizontal plane. Therefore, to train and predict using HIBER dataset (and also MMVR dataset), which consists of refined BBoxes as explained above, an additional module is required to convert the relaxed BBoxes predicted in the image plane into refined BBoxes. As a result, we modify RFMask in way that the BBox loss is calculated on the image plane and backpropagates to learnable parameters in end-to-end fashion. Specifically, we add an image BBox regression module alongside a horizontal BBox Regression module, enabling the conversion of BBox offsets to the image plane. By computing loss with respect to these offsets, we can learn refined BBoxes on the image plane. Additionally, the region proposals estimated by the region proposal network (RPN) are transformed into 3D BBoxes based on the fixed-height size, the same as the original RFMask. These BBoxes are then projected onto the image plane and 3D-to-2D projection.
F Definition of Metrics
Mean Intersection over Union: We adopt average precision on intersection over union (IoU) [6] as an evaluation metric. IoU is the ratio of the overlap to the union of predicted BBox A and annotated
BBox B as:
IOU (A, B) = IAN B
(22)
[AUB
Average Precision: Average Precision (AP) can then be defined as the area under the interpolated precision-recall curve, which can be calculated using the following formula:
AP
(r441 rc) Pantarp (re+1)
(23)
Pinterp (r)
max (r'), 2'2
(24)
19

---

Table Details of syper-parametery, Fixed-height size for HIBER dataset depend on the environ- ment.
Value
Name
Notation
P2S1 P2S2
WALK MULTI
training
190441 118280
58382 / 53690
# of validation
23899 33841
3229 / 8260
# of test
23458 85677
11931 8850
Data
Input radar healmap size
H x W
256 128
160
Segmentation mask size
X
x40x
x24x
Resolution of range
cm
11.5
12.2
Resolution of azimuth
deg.
1.3
1.3
Resolution of elevation Scale
deg.
1.3
1.3
log
Backbone
ResNet18
ResNet18
Total dimension positional embedding Ratio of depth dimension for TPE # of input frames
256
256
a
0.6
0.6
4
4 40x56
Extracted feature map size
His x w/s
64
Top-K selection
magnitude
magnitude
Model
Top-K
256
256
# of encoder blocks
Lau't
6
# of decoder blocks
Leross
6
6
# of head of multi-head atlention
M
4
4
queries
10
Threshold for detection and segmentation
0.5
0,5
Fixed-height height size (pixel)
H
36
Learning Transformation
True
False
Loss weight for GIoU horizontal plane
Abor
0.5
1.0
Loss weight GIoU on vertical plane
Aver
1.0
Loss weight for GIoU on image plane
Aaaage
CIoU
1.0
1.0
Loss weight L, on horizontal plane
Apar
0.5
1.0
Loss weight for L, on vertical plane
0.5
1.0
Loss weight for L, image plane
1.0
1.0
Batch size
32
32
Epoch for detection Epoch for segmentation
100
100
Training
20
20
Patience for early stopping
5
Check val every N epoch for early stopping
Optimizer
AdamW
AdamW
Learning rate
le-4
le-4
Sheduler
Cosine
Cosine
Maximum number of epachs for sheduler
100
100
Weight decay
1c-3
lc-3
# of workers
8
8
GPU (NVIDIA)
A40
A40
# of GPUs
1
Approximate training time
day
where The interpolated precision Pinterp at a certain recall level is defined as the highest precision found for any recall level r' 2 r. We present three variants of average precision: APs0. AP7s, and AP, where the former two represent the loose and strict constraints of IoU, while is the averaged score over 10 different loU thresholds in [0.5, 0.95) with a stepsize of 0.05.
Average recall (AR) [9] between 0.5 and of IoU overlap threshold can be computed by averaging over the overlaps of each annotation gl, with the closest matched proposal, that integrating over the recall axis the plot instead of the x IoU overlap threshold axis. Let be the IoU overlap and recall (o) the function. Let IoU (gt,) denote the IoU between
Average Recall:
20

---

Table 6: Results under "MULTT" on HIBER dataset.
Model
Dim
Input
BBox Loss
AP
APso
AP
AR1
AR10
DETR
2D
H
H+1 I
17.00
52.79
6.08
16.21
31.37
DETR (Top-K)
2D
H. V
H
22.74
60.64
11.66
18.62
37.62
RETR (TPE@Dec)
2D
H, V
H I
23.02
61.77
12.51
19.17
38.00
RETR
2D
H. V
H I
23.53
63.84
11.90
20.37
38.16
RETR (TPE@ Dec.)
3D
V
H+V+
26.36
69.50
14.33
20.76
40.90
RETR
3D
V
H+V+I
28.98
74.82
15.64
22.95
41.18
Table 7: Results under "P2S2" on MMVR dataset.
Model
Dim
Input
BBox Loss
AP
APso
AP7s
AR;
AR10
RFMask
2D
H. V
H ++1
6.03
22.77
0.88
9.25
12.09
DETR (Top-K)
2D
H, V
H ++I
9.29
34.69
2.49
20.68
22.82
RETR RETR
2D
H. V
H+1
10.37
35.40
3.36
18.83
20.06
3D
V
H V++
12.19
40.67
4.95
19.70
21.34
Table 8: Impact of Learnable Transformation (LT) with additional precision and recall metrics.
LT
AP
APso
AP7s
AR1
ARto
42.97
80.54
41.97
40.20
55.58
46.75
83.80
46.06
42.19
57.39
the annotation gt, and the closest detection proposal:
AR = [
recall(o)do
Σ
[IOU(tt)
0.5, 0)
(25)
Jo.5
The followings some variations of AR:
ARy: AR given detection per data.
AR1o: AR given 10 detection per data.
AR100: AR given 100 detection per data.
G
Additional Ablation Study
To validate the effectiveness of our RETR, we conducted additional ablation studies. Unless otherwise specified, the hyperparameters follow those listed in the Table 5.
Results under "MULTI" on HIBER dataset: Table 6 shows the evaluation results on the HIBER dataset. From this table, it can be seen that using RETR improves performance across all metrics. Similar to the MMVR results, RETR with trinplane loss shows enhanced performance. Additionally, use RETR with TPE both the encoder and decoder also contributes to performance improve- ments. However, compared to the "P2S1" of MMVR, the performance improvement is smaller (the improvement is 15.28 AP from DETR to RETR). This is likely because, unlike "P2S1", HIBER under "WALK" only involves walking actions, which benefit less from the of 3D information.
Results under "P2S2" on MMVR dataset: "P2S2" (Cross-Session and Unseen Split) on the MMVR dataset first splits all data segments in d5, d6, d7, and d9 into train, validation, and test sets. Then, is included all data in d8 in the test such that one can assess the generalization performance of trained model for an unseen environment (d8). Therefore, "P2S2" is the most challenging scenario the MMVR. Table shows the evaluation results under "P2S2" From the results the table, we confirmed that the prediction performance was improved by using RETR. In particular, RETR outperforms RFMask by margin of 11.92 + AP However, compared the results of P2S1, there significant decrease in performance, and this is due the unseen environment.
21

---

Table 9: Full table:
Tri-plane loss can improve the performance.
BBxx Loss
AP
AP
75
AR1
AR10
IoU
H+I 42.18
80.49
39.24
39.39
53.84
75.15
-+V+I
46.75
83.80
46.06
42.19
57.39
77.21
Table 10: Impact of the value of K on object detection
K
APso
AP7s
ARg
AR1o
64
38.39
80.00
31.85
37.95
50.79
100
38.82
80.06
32.99
38.85
52.28
196
46.97
82.65
45.55
42.93
57.62
256
46.75
83.80
46.06
42.19
57.39
Table 11: Impact of training data size on object detection
# of data
AP
APso
AP7s
AR;
AR.0
x0.1
37.10
78.30
31.30
37.50
50.33
x0.5
40.84
79.80
36.82
40.06
53.33
x1.0
46.75
83.80
46.06
42.19
57.39
Table 12: Inference time and frame rate (FPS) of RFMask and RETR.
Method
Time[ms]
FPS
RFMask
20.89
47.87
RETR 23.75
42.11
Impact of Tunable Dimension Ratio a TPE:
00
To investigate the impact of tuning in TPE, observed the performance differences when vary- ing the ratio of depth and angle dimensions (since the total dimension is dpoa 256, e.g..
84
55
82
ratio a 0.2 means depth is rounded to daep = adpos = 102 dimensions and angle to dang = (1 - ddge
Performance
AP
50
80
AP
154 dimensions). In Sec-
AP
45
tion 5.3, we showed detection results in Table 3b with selected some a and metrics. Here, show the results with more variants of a and metrics. Fig. 13 shows the result. The horizontal axis de- notes the proportion of depth dimensions, and the vertical axis denotes the performance of various metrics. Note that APso refers to the primary axis, while AP and AP7s refer to the secondary axis. The figure shows that the highest performance achieved when the depth proportion is a 0.6. The performance exhibits peak at this point, indicating that prioritizing depth improves performance.
0.0
0.1
0.2
0.7 Ratio of depth dimension
0.3
0.4
0.5
0.6
0.8
0.9
1.0
Figure 13: Full figure: Impact of tunable dimen- sion ratio in TPE.
Impact of Learnable Transformation: We expand Table 3b by including additional precision and recall metrics, providing a more comprehensive evaluation of the Learnable Transformation. The complete results are presented in Table 8.
Tri-Plane Loss:
In Section 4.5, Table 3c compared RETR with bi plane BBox loss (horizontal radar plane and image plane) to that with the tri-plane loss (including the vertical radar plane). We show the more results with complete metrics. The results in Table 9 highlight the necessity of accounting for the vertical BBox loss and the importance of leveraging features from the vertical radar heatmap.
22

---

Detection
Instance Segmentation
RETR
RFMask
GT
RETR
RFMask
GT
d55.3/002
d5s3/002
20
d6s4/009
07s1/000
d73
d8s5/007
d8s6/002
d9s1/000
d9s6/004
Figure 14:
Visualization and comparison between RETR and RFMask. Each row indicates the segment name used from the "P2S1" test dataset.
Impact of the Value of K in Top-K Selection: Table 10 shows that, as the value of K increases (e.g., K 196 and K 256), object detection performance improves across both precision and recall metrics.
Impact of Training Data Size: Table 11 reports the effect of training data size on detection performance using the MMVR dataset. We compare the original data size (x1.0) with 190, 441 radar frames against reduced data sizes of half (x0.5) and one-tenth (x0.1). The results demonstrate gradual improvement in detection performance as the data size increases.
Table 12 reports the average inference time in milliseconds, evaluated over all frames in the test data using an NVIDIA A40 GPU. RETR achieves an average inference time of 23.75 ms, which comparable to that RFMask at 20.89 ms.
Inference Time:
H Visualization Result
23

---

Detection
Instance Segmentation
RETR
GT
RETR
GT
d5x3/002
d5s4/007
d.566/00066
d6x4/009
d886//002 2202
d9s4/008
d9s6/004
Figure 15: Visualization of failure cases. Each row indicates the segment name used from the "P2S1"
test dataset.
Comparison: To compare the conventional RFMask with our RETR, we visualized the prediction results each. Fig. 14 shows these results. Each row indicates the segment name used from the "P2S1" test dataset. In detection, RFMask has some miss-detections, whereas RETR accurately predicts even when there are multiple subjects. For instance, RFMask tends to fail in detecting people close to the camera, as seen in d8s6/002, but this is improved with RETR. In segmentation, RETR captures human shapes more accurately than RFMask. However, RETR can also fail mask estimation, seen in the example of d9s6.
Analysis of Failure Cases: We provide failure cases in Fig. 15. As shown in images such as d5s3/002 and d6s4/009, RETR occasionally mispredicts bending-owe over person standing. Addi- tionally, as shown in d5s4/007, it is often challenging to predict the detailed position the arms, leading to failures in both detection and segmentation. In the segmentation mask region was excessively large or too narrow. Moreover, in instances such as d9s6/004, while the BBox prediction was successful, the segmentation failed. There also case, such as d9s4/005, where the inaccuracy in the BBox prediction led to an incorrect mask position.
cases, such d5s6/006 and d8s6/002,
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly outline the performance improvements in indoor perception achieved by the proposed algorithm using bullet points. Specifically, following each bullet point, Section 4 and Section 5 provide detailed descriptions of the proposed algorithm and its experimental results, demonstrating the effectiveness of the proposed method. Therefore, the overall content is consistent.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We included limitations and impacts in Section 5 and Section 6, respectively. We evaluated two datasets; MMVR and HIBER, and the properties of each dataset are described in Section 5 and Appendix E. Moreover, we analyzed the limitations by visualizing the failure cases in the Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
25

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provided the computational complexity in Appendix A. In the experiments, we provided mainly empirical results. However, by using two different datasets we showed that our proposed method is broadly applicable. In particular, we presented results using several types of tables and figures.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We included in Appendix E all the important hyperparameters we used in our experiments. We also described the architecture design we built in detail in Appendix A. In addition, we included split of the dataset in Appendix E as well. This allows us to reproduce the main experimental results in the paper. Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
26

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We plan release the source code online. And the link the datasets used in the experiment was provided in Section 5.1.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips.ccpput wttpcccppppcccceeee ctllic//giieee des/ CodeSubmi ssisnnl ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We included details on their hyperparameters and data. See Table for details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the large computational complexity, statistical results are not included. However, we reported more comprehensive experimental results by conducting experiments on multiple datasets and under multiple conditions. See Section and Appendix for details.
27

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Please refer Table 5, which specifies computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https :ttttpp://euuriiie /tteueuurpp..ee wtwpeurrpssccppplleeeeeepe httpp///ccctcchtcccccceeeeeeeeee
didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification:
We carefully confirmed it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
28

---

Justification:
We discussed social impacts in Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: In paper, we only evaluated datasets that distributed through regular procedures. We also did not use pre-training model, so there is no such a risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual- use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Each of papers: MMVR and HIBER datasets we used, properly cited; see that each paper is cited in the dataset explanation in Section 5.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
29

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: released the source code and detailed experimental conditions required to reproduce the experiments are included in the Appendix E.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
30

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "vPTxFFyfzf", "reviewer": "Reviewer_DJHb", "summary": "The paper introduces Multi-View Radar Detection Transformer for indoor object detection. Inspired from DETR, the authors propose and end-to-end RETR to detect objects from the radar inputs. To improve the feature association across the two radar views, the authors introduce a new Tunable Positional Embedding. The proposed approach achieves the solid performance on the standard benchmarks of radar-based indoor object detection.", "questions": "Please refer to my weakness section.", "limitations": "The authors have discussed the limitations and broader impact in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The proposed approach is simple yet well-motivated. The proposed RETR eliminates the cumbersome design of the detection network for radar inputs. \n- The analysis of Tunable Positional Embedding Section 4.3 is comprehensive and well-motivated.\n- The proposed method achieves strong experimental results on MMVR and HIBER datasets.", "weaknesses": "-  I am wondering how the Top-K Feature selection impacts the performance of the network. How can we choose the number of K? It seems that there is no experiment to validate the choice of K. It will be better if the authors conduct an ablation study to explore the effectiveness of choosing K.\n- What is the computation cost of the proposed RETR model? It will be better if the authors report the inference time/computational cost of the proposed model\n-  For the learnable radar-to-camera coordinate transformation, while I acknowledge the performance improvement according to this proposed model, I have several questions related to this module. \n- - Does the entire dataset use the same set of learnable vector $\\omega$ and translation vector $t$? Or each radar sample in the data will have a different vector $\\omega$ and translation vector $t$?\n- - How can we verify that the learnable radar-to-camera coordinate transformation is accurate? Is there any way to evaluate it using the ground truths?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments and valuable feedback! We provide our point-to-point responses below.\n\n**I am wondering how the Top-K Feature selection impacts the performance of the network. How can we choose the number of K? It seems that there is no experiment to validate the choice of K. It will be better if the authors conduct an ablation study to explore the effectiveness of choosing K.**\n\nWe appreciate Reviewer DJHb's suggestion. In response, we have conducted additional experiments to examine the impact of $K$ on detection performance. The results, detailed in Table 1 of the PDF in our global response, indicate that increasing $K$ improves object detection performance. However, as noted in our detailed response below to your next comment on the computation cost and inference time, choosing a larger $K$ also significantly increases the training and inference time.\n\n**What is the computation cost of the proposed RETR model? It will be better if the authors report the inference time/computational cost of the proposed model.**\n\nThanks for pointing out the computation cost and inference time. \n\nFirst, following the computational complexity notation used in the DETR paper, every self-attention mechanism in the encoder has a complexity of $\\mathcal{O}(d^2 2K + d (2K)^2)$ where $d$ is the embedding dimension and $K$ is the number of selected features from the Top-$K$ selection. The cost of computing a single query/key/value embedding is $\\mathcal{O}(d' d)$ (with $d=Md'$ where $M$ denotes the number of attention heads and $d'$ the dimension in each head), while the cost of computing the attention weights for one head is $\\mathcal{O}(d' (2K)^2)$. Other computations may be negligible. In the decoder, each self-attention mechanism has a complexity of $\\mathcal{O}(d^2 N + d N^2)$ where $N$ is the number of queries, and the cross-attention between query and multi-view radar features has a complexity of  $\\mathcal{O}(d^2(N + 2K) + d 2NK)$. \nIn conclusion, the overall complexity of our RETR model is $\\mathcal{O}(4d^2 K + 4d K^2 + 2d^2 N + d N^2  + 2d NK)$.\n\nSecond, regarding the inference time, we report the average inference time in milliseconds in Table 3 of the PDF in the global response. We used an NVIDIA A40 GPU to evaluate the inference time over all frames in the test data. RETR achieved an average inference time of $23.75$ ms that is comparable to $20.89$ ms of RFMask. \n\n**For the learnable radar-to-camera coordinate transformation, while I acknowledge the performance improvement according to this proposed model, I have several questions related to this module.\nDoes the entire dataset use the same set of learnable vector and translation vector ? Or each radar sample in the data will have a different vector and translation vector ?**\n\nThe learned vectors $\\omega$ and $t$ (or, equivalently, the rotation matrix $R$ and translation vector $t$) are fixed and applied consistently to all test frames.  \n\nDuring training,  $\\omega$ and $t$ were updated from one minibatch to the next, as they are part of the learnable parameters in RETR.\n\n**How can we verify that the learnable radar-to-camera coordinate transformation is accurate? Is there any way to evaluate it using the ground truths?**\n\nGood question. One way to verify the coordinate transformation learning is, for a given point in the 3D radar coordinate, by checking the distance between the two transformed points using calibrated and learned coordinate transformation. Although the calibrated coordinate transformation (including both rotation matrix and translation vector) is NOT ground truth due to radar resolution (also see our responses to Reviewer XrXa), it serves as a reasonable baseline or reference benchmark for the learned coordinate transformation.  \n\nIn the PDF of the global response, Fig.1 shows the distance difference between the two transformed points over the training steps using the MMVR dataset. We randomly initialize the learned rotation matrix and translation vector at iteration 1. The results demonstrate that, as training progresses, the learned radar-to-camera coordinate transformation becomes increasingly aligned with the calibrated one, indicating that the learning is moving in the correct direction. \n\nFinally, we'd like to add that by using the learnable radar-to-camera coordinate transformation, it is possible to incorporate the radar-to-camera geometry into the end-to-end radar perception pipeline without the need for a cumbersome calibration step, while still achieving comparable perception performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jMgxqOY6x9", "reviewer": "Reviewer_H5qV", "summary": "In this paper, the authors propose Radar dEtection TRansformer (RETR), an extension of the popular DETR architecture, tailored for multi-view radar perception. RETR inherits the advantages of DETR, eliminating the need for hand-crafted components for object detection and segmentation in the image plane.", "questions": "Is there any interference (cross-path signals) between the two radars deployed in vertical and horizontal directions?", "limitations": "Usually a large amount of high quality radar data is required to train the transformer. It is highly recommended to carry out validation with different amount of training data.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Two radars with different directions are deployed. Transformer is applied for segmentation.", "weaknesses": "For indoor perception with radar, multi-path is expected. It would be good to add a paragraph to discuss this issue and how to mitigate multi-path or its impact to the perception/segmentation performance. \n\nThe motivation of applying transformer for indoor segmentation is not well discussed. What are the benefits and unique challenges of applying transformer to radar based indoor segmentation?\n\nFor the introduction of Generation of Radar Heatmaps, please consider cite the following paper:\nS. Sun, A. P. Petropulu and H. V. Poor, \"MIMO Radar for Advanced Driver-Assistance Systems and Autonomous Driving: Advantages and Challenges,\" in IEEE Signal Processing Magazine, vol. 37, no. 4, pp. 98-117, July 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments and valuable feedback! We provide our point-to-point responses below. \n\n\n**For indoor perception with radar, multi-path is expected. It would be good to add a paragraph to discuss this issue and how to mitigate multi-path or its impact to the perception/segmentation performance.**\n\nThanks for the insightful comment. We agree that multi-path reflections from the ground, ceiling, and other strong scatterers (e.g., metal) can cause (first-order or second-order) ghost targets and elevate the noise floor. One way to address this issue is to employ classical signal processing techniques into the radar heatmap generation to remove these ghost targets and the static background reflection. \n\nWe may address this issue directly in the end-to-end radar perception pipeline by labeling these ghost targets in standard radar heatmaps (although it is difficult and costly to have these labels) and directly classifying RETR object queries to one of {$\\{\\emptyset, person, ghost\\}$}, alongside regressing queries to the bounding box parameters. As you suggested, we will add a paragraph to discuss the multi-path issue and potential ways to mitigate its impact.\n\n**The motivation of applying transformer for indoor segmentation is not well discussed. What are the benefits and unique challenges of applying transformer to radar based indoor segmentation?**\n\nWe agreed that in the main paper, we primarily used the object detection example to motivate the RETR, while the segmentation part was deferred to Appendix B Segmentation. In the updated paper, we will emphasize that the segmentation is an integral part of the RETR by highlighting the following points in the main paper:\n\n1. The segmentation head uses the estimated bounding box (BBox) as a prior or constraint to classify each pixel within the BBox (see Fig. 8 Illustration of Segmentation Head in Appendix).\n   \n2. The pretrained RETR components, such as the backbone, top-$K$ selection, and the detection transformer are reused with frozen weights to train the segmentation head (the lower branch) in Fig. 3 of the main paper.\n\nWe will point out the challenges in extracting finer-grained radar features, fusing features from two radar views, and utilizing the prior BBox from the detection head to support the pixel-level segmentation task. We will emphasize how to address these challenges by leveraging the DETR architecture to avoid cross-view radar feature association and introducing additional modifications, such as tunable positional embedding, radar-to-camera coordinate transformation, and tri-plane loss.\n\n**For the introduction of Generation of Radar Heatmaps, please consider citing the following paper: S. Sun, A. P. Petropulu and H. V. Poor, \"MIMO Radar for Advanced Driver-Assistance Systems and Autonomous Driving: Advantages and Challenges,\" in IEEE Signal Processing Magazine, vol. 37, no. 4, pp. 98-117, July 2020.**\n\nThank you for the suggestion. We will add the suggested paper to the reference list.\n\n**Is there any interference (cross-path signals) between the two radars deployed in vertical and horizontal directions?**\n\nThanks for raising this important point. To prevent cross-radar interference between the horizontal and vertical radar sensors, the two radars were configured to operate at different frequency bands. In the HIBER dataset, the horizontal radar operates in the $77 - 78.23$ GHz band, while the vertical radar is in the band of  $79-80.23$ GHz. In the MMVR dataset, the horizontal radar operates in the $77-78.36$ GHz band, while the vertical radar is in the band of  $79-80.36$ GHz. In both cases, there is a minimum gap of $500$ MHz between the operating frequency bands of the two radars.\n\n**Usually a large amount of high quality radar data is required to train the transformer. It is highly recommended to carry out validation with different amount of training data.**\n\nGood point. In Table 2 of the PDF in the global response, we report the impact of training data size on detection performance using the MMVR dataset. We compare the original data size (x1.0) with $190,441$ radar frames against reduced data sizes of half (x0.5) and one-tenth (x0.1).  The result shows a gradual improvement in detection performance with an increase in data size, particularly at higher IoU thresholds, such as AP$_{75}$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "valYQIc2j1", "reviewer": "Reviewer_kPes", "summary": "The primary content of this paper is an introduction to a multi-view radar detection transformer algorithm (RETR) for indoor perception. The algorithm achieves effective object detection in indoor environments by utilizing multi-view radar data and combining self-attention mechanisms and cross-attention mechanisms. The author validates the performance improvement of the algorithm through experiments and discusses its application in indoor perception.", "questions": "1.Could you provide a detailed description of the design of the Top-k selector?\n2.Can you provide a more detailed analysis of the impact of the proposed RETR on real-time performance?", "limitations": "NA", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The idea is helpful for the field of view indoor multi-view radar perception.\n2.The paper is clear and easy to follow. \n3.Rigorous ablation studies were conducted, providing evidence of the proposed method's efficacy.\n4.RETR, based on the original DETR, has achieved significant performance improvements on indoor multi-view radar datasets with heatmap input.", "weaknesses": "1.Radar perception datasets primarily utilize point clouds as the data form. Methods based on heatmaps for indoor multi-view radar perception are not yet sufficient. The author mainly compares with customized DETR and RFMask, but the comparison is insufficient. Have the authors considered conducting further comparison experiments on the HUPR [1] dataset? If the authors can further demonstrate the method's generalization ability, I would consider giving a higher score.\n2.The method presented is an improvement on the DETR and combines existing methods in a novel way. While innovative, it's not highly original for NeurIPS.\n3.Research on indoor multi-view radar perception using heatmaps is relatively scarce. The author is unable to provide source code, potentially limiting contributions to this field.\n\n[1] Lee, Shih-Po, et al. \"Hupr: A benchmark for human pose estimation using millimeter wave radar.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments and valuable feedback! We provide our point-to-point responses below. \n\n**Radar perception datasets primarily utilize point clouds as the data form. Methods based on heatmaps for indoor multi-view radar perception are not yet sufficient. The author mainly compares with customized DETR and RFMask, but the comparison is insufficient.\nHave the authors considered conducting further comparison experiments on the HUPR [1] dataset? If the authors can further demonstrate the method's generalization ability, I would consider giving a higher score.**\n\nBorrowing Table 1 from the MMVR paper, also listed as Table 4 in the PDF of the global response, you are correct that indoor radar perception datasets primarily use point clouds. However, an increasing number of datasets employ multi-view radar heatmaps to support more diverse perception tasks: RF-Pose, HuPR, HIBER, and MMVR, with the latter three collected since 2022. RF-Pose and HuPR have a resolution of $15^\\circ$ in the two-view radar heatmaps, while HIBER and MMVR offer a finer resolution of $1.3^\\circ$.\n\nThanks for pointing out the HuPR dataset. First, HuPR focuses on `pose estimation (keypoints)`, while our work targets `object detection (BBox estimation)` and `segmentation (pixel-level masks)`. To utilize the HuPR dataset for training and evaluating our RETR pipeline, we would need to extract bounding box and segmentation labels from each HuPR frame which, unfortunately, cannot be completed within the rebuttal period.\n\nSecond, even with BBox and mask labels from HuPR, our RETR pipeline requires geometric information about the radar and camera coordinate systems, including the radar-to-camera transformation (rotation matrix and translation vector) and the 3D-to-2D camera projection matrix (pinhole camera model). Both HIBER and MMVR datasets provide this calibrated information. While our learnable radar-to-camera transformation can reduce some geometry dependency, the 3D-to-2D projection matrix is still necessary. `We have reached out to the HuPR authors for this additional geometric information.` Once we receive it, we will report the RETR performance on at least BBox estimation.\n\nThird, we believe that the proposed RETR pipeline and its evaluation over `two separate datasets` (HIBER and MMVR) have sufficiently demonstrated the generalization capability. It is noted that the radar-to-camera geometries in these datasets are completely different but known (via calibration or learning). RETR is purposely designed to handle these differences by incorporating calibrated/learnable radar-to-camera transformations, 3D-to-2D projections, and tri-plane loss functions.\n\n\n**The method presented is an improvement on the DETR and combines existing methods in a novel way. While innovative, it's not highly original for NeurIPS.**\n\nWe believe our contributions to be novel and original as they are not a simple combination of existing methods. Indeed it is true that we build upon DETR, yet we propose several contributions that make our RETR unique. First, we highlight how we adapt DETR to the multi-view scenario thanks to the self-attention mechanism combined with the Top-K selection and by reusing the cross-attention mechanism to avoid traditional object-to-multi-view-feature association. \n\nSecond, we introduce a depth-prioritized feature similarity via a tunable positional embedding (TPE), incorporating a crucial inductive bias of shared depth across the two radar views to enhance downstream tasks.\n\nThird, we propose a tri-plane loss from both radar and camera coordinate systems which, to the best of our knowledge, has never been considered for object detection from radar heatmaps.  \n\n**Research on indoor multi-view radar perception using heatmaps is relatively scarce. The author is unable to provide source code, potentially limiting contributions to this field.**\n\nWe plan to release our code after the paper acceptance.\n\n**Question 1: Could you provide a detailed description of the design of the Top-k selector?**\n\nRegarding the Top-$K$ selection, we begin with the features map extracted from the shared backbone. Each cell in the feature map can potentially be considered as a patch/token for input into the subsequent transformer encoder. To alleviate the time complexity of the attention module, we select only the tokens with the highest norms computed over the channel dimension. In this way, we propagate only the most relevant information to the following modules and increase the inference performance, while keeping the complexity low.\n\nIn Fig. 6 of the main paper, we visualize selected top features (and their locations in the radar heatmap) in the two radar views that are used to detect bounding boxes in the image plane by inspecting the cross-attention module. \n\nIn Table 1 of the PDF in the global response, we have included an additional ablation study examining the impact of $K$ on detection performance.\n\n**Question 2: Can you provide a more detailed analysis of the impact of the proposed RETR on real-time performance?**\n\nWe report the inference time in Table 3 of the PDF in the global response. We used an NVIDIA A40 GPU for evaluation. RETR has a comparable inference time as RFMask ($20.89$ ms against $23.75$ ms of RETR). We compute the average inference time across all radar frames in the test data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FRBsCxyBlu", "reviewer": "Reviewer_XrXa", "summary": "This work introduces a novel method RETR for indoor object detection and segmentation based on multi-view radar heatmaps. RETR extends the popular DETR framework and incorporates modifications specific to multi-view radar perception, such as depth-prioritized feature similarity via TPE, a trip-plane loss, and a learnable radar-to-camera transformation. Experiments on two datasets demonstrate the effectiveness of RETR for object detection and segmentation.", "questions": "1. At line 28, it is better to explain why radar heatmaps are more preferred for challenging perception tasks. At line 31, it's not necessary to mention if one work is publicly accessible for the introduction section.\n2. The relationship and different between previous works in radar-based detection and segmentation, e.g., RFMask and RETR is not well explained.\n3. Better to show the tokenization step in Figure 3 to aid in understanding the pipeline.\n4. In line 135, equation 3 should be Equation 3.\n5. How did the author implement the top-K feature selection step? And how to supervise such a selection step to ensure the selected features are significant for the detection task?\n6. In line 229, what's the reason why the author claimed that the calibration may be only accurate for a limited interval of depth and angles.\n7. The MMVR dataset can not be found online.\n8. Please provide captions for table 1 and 2 to increase the readability.", "limitations": "NA.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work utilizes multi-view radar heatmaps as input for indoor human perception, which has broad application for privacy-aware indoor sensing and monitoring.\n2. This work extends image-based DETR to multi-view radar-based RETR, establishing a new baseline for radar-based human detection and segmentation.  The proposal of tunable positional embedding is interesting. It can enable the adjustment of different axes for positional embeddings. \n3. The experiments are comprehensive with two tasks validated on two tasks. The details of hyperparameters and implementation details are adequate, making this paper a good reference for future works.", "weaknesses": "1. Most commercial radars have 2D virtual antenna arrays, e.g., 16*8, instead of only a pair of 1D antenna arrays. Is there any specific reason for considering such kinds of radar antenna arrays? Normally, the 2D heatmaps (range-elevation, range-azimuth) are generated via projecting the 3D radar cube data in two views. The explanation in Section 2 does not fit the real process of multi-view radar heatmap generation. If this work considers perception with such a unique radar antenna array, better to introduce what types of radar the dataset used.\n2. In the experiments, only radar heatmap-based methods are used for comparison. Better to incorporate the results of radar point cloud-based and camera-based approaches for a more complete comparison. The survey of related work is also not complete, ignoring recent works in radar-based object detection for autonomous driving [1-3], which also use radar heatmaps:\n[1] Liu, Yang, et al. \"Echoes beyond points: Unleashing the power of raw radar data in multi-modality fusion.\" Advances in Neural Information Processing Systems 36 (2024).\n[2] Paek, Dong-Hee, Seung-Hyun Kong, and Kevin Tirta Wijaya. \"K-radar: 4d radar object detection for autonomous driving in various weather conditions.\" Advances in Neural Information Processing Systems 35 (2022): 3819-3829.\n[3] Skog, Mikael, et al. \"Human Detection from 4D Radar Data in Low-Visibility Field Conditions.\" arXiv preprint arXiv:2404.05307 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments and valuable feedback! We provide our point-to-point responses below. Due to space constraints, we have shortened your comments for brevity.\n\n**Most commercial radars have 2D virtual antenna arrays, e.g., 16*8, instead of only a pair of 1D antenna arrays. Is there any specific reason for considering such kinds of radar antenna arrays?**\n\nThanks for raising this point. The main reason for using a configuration of two cascading radars in horizontal and vertical orientations is to harness `finer angular resolution` in both azimuth and elevation domains to support all three perception tasks, especially for pixel-level segmentation. Both HIBER and MMVR datasets used this configuration. \n\nMost commercial radars have a typical configuration of $4$ Rx and $3$ Tx antennas, yielding a virtual array of $8$ elements in one angular dimension and $4$ in the other. Examples include TI's IWR1443 and IWR1843 chipsets and, more recently, NXP's TEF81xx and TEF82xx series.  This usually leads to an angular resolution of $15^\\circ$ in one angular dimension and about $30^\\circ$ in the other angular dimension. \n\nThe configuration of two cascading radars ($12$ Tx and $16$ Rx) yields a virtual array of $86$ non-overlapping half-wavelength-spaced elements in both vertical and horizontal dimensions, offering an angular resolution of $1.3^\\circ$, more than $10 \\times$ better. The resulting high-resolution multi-view radar heatmaps can provide fine-grained radar features and support not only BBox estimation and pose estimation but also the more challenging pixel-level segmentation. \n\n**The explanation in Section 2 does not fit the real process of multi-view radar heatmap generation. Better to introduce what types of radar the dataset used.**\n\nDue to space constraints, the current version of Section 3 - Generation of Radar Heatmaps presents an abbreviated description of the full process by skipping steps such as MIMO waveform separation for virtual array processing, integration over the Doppler domain, and projection onto the azimuth and elevation domains. In the updated paper, we plan to include a new section in the Appendix to introduce the dual (horizontal-vertical) radar configuration and provide a detailed explanation of the generation of the two radar heatmaps.\n\n**In the experiments, only radar heatmap-based methods are used for comparison. Better to incorporate the results of radar point cloud-based and camera-based approaches for a more complete comparison.**\n\nThanks for the suggestion. We have two response points: \n\n1. `Dataset`: To the best of our knowledge (see Table 4 in the PDF of the global response), we cannot find an indoor radar dataset with both radar heatmap and point cloud formats for evaluating a given method (either RETR or a baseline). \n\n2. `Baseline`: We have included DETR, a camera-based method, in our baseline comparisons. We will make an effort to incorporate a radar cloud-based baseline in the updated paper.\n\n**The survey of related work is also not complete, ignoring recent works in radar-based object detection for autonomous driving [1-3], which also use radar heatmaps: [1] Liu, Yang, et al... [2] Paek, Dong-Hee, ... [3] Skog, Mikael....**\n\nThanks for pointing out these recent radar datasets featuring radar heatmaps. We will include them in the updated paper. Specifically, we plan to discuss [3] in Section 2 - Related Work, as it is particularly relevant to our task of human perception using radar heatmaps.\n\n**Line 28: better to explain why radar heatmaps are more preferred for challenging perception tasks.**\n\nIn the updated paper, we plan to highlight the difference between radar heatmaps and point clouds. We will also explain how high-resolution, multi-view radar heatmaps may be better suited for supporting more challenging perception tasks, such as pixel-level segmentation.\n\n**The relationship and difference between previous works in radar-based detection and segmentation...is not well explained.**\n\nIn Fig. 2 of the main paper, we intended to highlight the major differences between RFMask and the proposed RETR. \nRFMask uses regional proposals and features from only the horizontal radar view with a fixed height. RETR, on the other hand, employs a detection transformer, fuses multi-view radar features, and exploits the unique multi-view radar setting via TPE and radar-to-camera coordinate transformation. We will expand the `RFMask with Refined BBoxes` section in the Appendix to clarify these differences further.\n\n**Better to show the tokenization step in Figure 3....**\n\nsee Fig. 2 of the PDF in the global response.\n\n**implement the top-K feature selection? And how to supervise...?**\n\nFor the Top-$K$ selector, we just select feature tokens with the highest norms computed over the channel dimension. We observed that direct supervision is unnecessary for this step because radar features tend to be extremely localized. Please refer to Fig. 6 of the main paper for visualizing the cross-attention between selected features and detected BBoxes. \n\n**Line 229, why the author claimed that the calibration may be only accurate for a limited interval of depth and angles.**\n\nThis is mainly due to the varying cross-range radar resolution over depth and angles as the radar operates in a polar coordinate system.  For a given angular resolution, the cross-range cell resolution at $3$ meters will be about $3\\times$ larger than that at $1$ meter. To compensate for such resolution differences, one may need to repeat the calibration for different depth/angular intervals.   \n\n**The MMVR dataset can not be found online.**\n\nWe reached out to the MMVR authors who provided us with the \"P2\" split at the time of submission. \nThe MMVR dataset should be available now by searching \"MMVR Dataset\", although we are not allowed to share any links in the rebuttal.\n\n**Minor questions: Line 31:... Line 135: .....provide captions for table 1 and 2** \n\nWe will make the changes accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful comments, suggestions and questions. In the rebuttal form to each reviewer, we provide detailed point-to-point responses. 

In these point-to-point responses, we often refer to the attached PDF for additional results in the form of tables and figures.

Meta-review: The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. All reviewers recommend acceptance after discussion (with three weak accepts and one accept), and the ACs concur. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.
Decision: Accept (poster)