TARSS-Net: Temporal-Aware Radar Semantic Segmentation Network
Authors: Youcheng Zhang
Liwen Zhang
ZijunHu
Pengcheng Pi
Teng Li
Yuanpei Chen
Shi Peng
Zhe Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: Radar Semantic Segmentation
Temporal Relation Modeling
Primary area: machine_vision
TL;DR: Discussions on current temporal modeling mechanisms; a novel temporal information learning paradigm for radar semantic segmentation.

Abstract: Radar signal interpretation plays a crucial role in remote detection and ranging. With the gradual display of the advantages of neural network technology in signal processing, learning-based radar signal interpretation is becoming a research hot-spot and made great progress. And since radar semantic segmentation (RSS) can provide more fine-grained target information, it has become a more concerned direction in this field. However, the temporal information, which is an important clue for analyzing radar data, has not been exploited sufficiently in present RSS frameworks. In this work, we propose a novel temporal information learning paradigm, i.e., data-driven temporal information aggregation with learned target-history relations. Following this idea, a flexible learning module, called Temporal Relation-Aware Module (TRAM) is carefully designed. TRAM contains two main blocks: i) an encoder for capturing the target-history temporal relations (TH-TRE) and ii) a learnable temporal relation attentive pooling (TRAP) for aggregating temporal information. Based on TRAM, an end-to-end Temporal-Aware RSS Network (TARSS-Net) is presented, which has outstanding performance on publicly available and our collected real-measured datasets. Code and supplementary materials are available at https://github.com/zlw9161/TARSS-Net.

TARSS-Net: Temporal-Aware Radar Semantic Segmentation Network
Youcheng Zhang"* 1 Liwen Zhang
Zijun Hu' Shi Peng'
Pengcheng Pi¹ Zhe Ma-1
Teng Li Yuanpei Chen'
Intelligent Science and Technology Academy of CASIC
SShenzhen International Graduate School, Tsinghua University youcheng178166
1wzhang916 (eeeee66611111.11111 e126. com" mazhe_ tatihe_theee63..om
Abstract
Radar signal interpretation plays crucial role in remote detection and ranging. With the gradual display of the advantages of neural network technology in sig- nal processing, learning-based radar signal interpretation is becoming research hot-spot and made great progress. And since radar semantic segmentation (RSS) can provide more geneggrined target information, has become a more concerned direction in this field. However, the temporal information, which is an important clue for analyzing radar data, has not been exploited sufficiently in present RSS frameworks. In this work, we propose novel temporal information learning paradigm, i.e., data-driven temporal information aggregation with learned target-history relations. Following this idea, flexible learning module, called Temporal Relation-A ware Module (TRAM) is carefully designed. TRAM contains two main blocks: i) encoder for capturing the target-history temporal relations (TH-TRE) and ii) learnable temporal relation attentive pooling (TRAP) for aggre- gating temporal information. Based on TRAM, end-to-end Temporal-A ware RSS Network (TARSS-Net is presented, which has outstanding performance on publicly available and our collected real-measured datasets. Code and supplemen- tary materials are available htttss//gihubb.oo/2///11////
Introduction
Radar is reliable remote sensing device for its robustness in adverse weather and illumination
conditions. It is widely used for applications like the autonomous driving [33 21 36), Unmanned Aerial Vehicle (UAV) surveillance [ [9] sea monitoring 31 24 32), etc. However, the received scattered radar signals are high-entropy information body coupled with environmental clutters from large spatial range, device noise and moving target information (i.e., distance/range, direction/angle and velocity/Doppler shift). These components are additionally condensed into the received echoes, which is inherently a gap in understanding of radar signals to human perception, opposed to visible light images, which are modalities aligned with human visual perception, and natural language texts that are naturally based on human semantic understanding. This human-unfriendly perception gap causes great difficulty semantic perception of scenes and objects 22 signals and processing is in Appendix
More information on radar
Encouraged by the success of deep learning techniques in computer vision, in especial, object detection [ 17 27 26 and semantic segmentation 20
28
2 10
some efforts have been made
recently better understand complex radar data. Compared with detection models [3 using
bounding boxes, the segmentation models [ 14
36 35 can provide pixel-wise (a unit of the
"Equal contributions.
1 Cooresponniig author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

range-angle (RA) or range- doppler (RD) frequeney representation) detection results for objects and even the background, which is practical for radar scene understanding and is also followed by this seencodingddeoodiin (CAED), which
work. Most of these methods are based on convolutional take the frequency representations of series of fast Fourier transforms (FFT's on the radar signals as input, and make predictions on the RA or RD view or both views.
By analysing current efforts in RSS, two primarily concerned problems can be concluded for effective design of RSS system. i) How to capture target signature effectively in spatial domain. Typically, atrous spatial pyramid pooling (ASPP) 3 is used to obtain multi-scale spatial information [14] 21] and deformable convolution is utilized to extract target features with irregular shape [o] Specifically, PeakConv (PKC) [35] the learning- based operator tailored for radar signals which aims to capture target signature's 's salience. ii) How to utilize inherent temporal information of radar input tensor. is worth noting that, compared with the direct use of existing methods, taking into account the intrinsic merits of radar signal can achieve more performance advantages. In terms of temporal information utilization, the common practice is still 3DConv. Few work has explored specific temporal modeling mechanism for RSS and even less conducted discussion of other off-the-shelf methods from the perspective of RSS task.
Commonly, 3D convolution (3DCCnv) is implemented [21 33 6
Considering the above research gaps, this paper conducts a comprehensive discussion from the perspective of RSS for existing time series modeling paradigms. including i) causal temporal relation modeling. represented by hidden Markov models (HMM) (25) and recurrent neural network (RNN) family 7 4: ii) parallelized sequence representation modeling, represented by 3DConv (2] and Transformer (30 family. Based on this, we propose a novel modeling paradigm for radar temporal information utilization. Specifically, this paper makes the following efforts:
i. A novel temporal modeling paradigm for RSS. Based on the in-depth analysis and discussion of existing temporal modeling methods, several specific design principles for RSS temporal modeling paradigm are given ($ ($2). Following these principles, a novel temporal information learning paradigm for radar spatio-temporal tensors proposed, i.e., data-driven temporal information aggregation with learned parget-history relations.
ii. Temporal relation attentive module (TRAM). TRAM is proposed to realize the learning paradigm mentioned above, which is a flexible temporal relation encoding and aggregating module. For encoding, the target-history temporal relation encoder (TH-TRE) is designed, which aims to capture relations between target frame and its historical neighbors. For aggregating, temporal relation attentive pooling (TRAP) with two forms, i.e., learning in temporal-depth and spatio-temporal, is presented. Both the encoder and aggregator can be independently inserted into arbitrary 2/3DConv- based networks ($
ii.. Reasonable-scale RSS Network with variable input time lengths. Processing high- nigh-dimensional radar sequence while ensuring the use of temporal information is computationally expensive. To ensure RSS model has the ability of online tile processing, temporal-aware RSS network (TARSS-Net) with moderate parameters is designed based on TRAM, which keeps its parameter scale constant while accepting adjustable time length input. Compared with existing temporal modeling methods, TARSS-Net could effectively avoids the long-term dependence decreasing risk well as gradient disappearance problem, which shows good segmentation performance and inference efficiency in both multi-view and single-view conditions ($ 3 Appendix C).
iv. Model performance verification with real-measured data in different detection scenarios. To verify the scope of application of TARSS-Net, we conduct quantitative experiments on different real-measured large scale radar datasets including CARRADA [22] which is collected from a low cost FMCW (s= 77GHz) on-board radar in driving scenario and self-collected dataset, KuRALS, recorded from a Kurz-under (Ku) band (+= 17GHz) radar for UAV surveillance and sea monitoring. Experimental results show TARSS-Net can achieve state-of-the-an (SoTAA performance ($ 4..
2 Discussion of Temporal Modeling Paradigms for RSS
In this section, a deep discussion and analysis of current temporal modeling paradigm is conducted. On this basis, the design principles of spatio-temporal encoding suitable for RSS domain are given, and then TRAM paradigm is developed. For better explanation, Fig. [ intuitivell presents the core modeling mechanism of the discussed paradigms.
2

---

Obseevation
Leamed Repecasstation
Tidden
W: Taok-rehbeed Contriutioon
Prrameter fies cornection
Pasometertzedd connection
(a1) HLMM (probabilisti dependence)
pedding
(b1) C Convolation dlocal contextaal leaming!
X..
X
X
X
(a2) RNN Osommable vectorized dependence)
(M] Self-attention (dense prir-wis relation)
X
(a) temporal relntion modeling
(b) Parallelized sequence representation modeling
(c) (aages-histaryy relation modeling)
Figure I: Brief illustration of temporal modeling paradigms.
i. Causal temporal relation modeling.
is the classic causal temporal modeling methods, which uses hidden states model temporal dependencies as shown in Fig. (al). Based on the assumption of Markov first-order homogeneity and observation independence. can conveniently use the transition probabilities between hidden states to describe the intrinsic causal relationship of sequential data. However, shallow probabilistic model, HMM cannot perform data representation and downstream task prediction end-to-end. Meanwhile, the Markov assumption and discretized encoding of the hidden states also limit its ability to describe long-term dependencies.
HMM 25
RNN & also introduces hidden state deseribe temporal relation of input sequence similar with HMMs. However, its hidden state is extended to a continuous vector, i.e., real-valued representation projected by a parameterized connection shown in Fig. [a()). In this way, RNN is no longer limited to a probabilistic model, but a learnable one that can be deepened. The gradient descent algorithm can be used realize end-to-end training of RNN's representation and prediction, which greatly alleviates the problem of HMM's insufficient ability to depict long-term dependencies. With more complex computing units such as LSTM [2 7
and GRU 44, its representational power can be further enhanced. However, in addition to the resistance caused by gradient dilpersionndisspppaaaacc under the back- sock-popagattio framework, RNNs are sequential causal computing model after all, and their popularity is gradually fading under current main theme of multi-core parallel computing framework.
ii Parallelized sequence representation modeling.
3DCony 29 uses the concept of local receptive field (LRF) and the design of shared kernel. can efficiently fulfill encoding and representation of fiigh-dimnsionall spatio-temporal tensors in a multi-core computing environment as shown in Fig. (b1). Therefore, 3DConv has become the preference for RSS models that require spatio-temporal encoding capabilities [14] 21 33 O However, due to the limitation of LRF, it cannot achieve range while retaining efficient computation and appropriate parameter amount. Meanwhile, an inherent contradiction between 3DConv and RSS is ignored by most of current works, that is, convolution naturally follow the rules of context encoding, which is more suitable for obtaining responses the center position of LRF. For input sequence, {X1, ... X1+r: .X122> e H (2r+1) performing predictions on X1 is more of a natural advantage 3DConv-base models, instead of predicting X1-2r most RSS required. In terms of pragmatism, performing the prediction at the current instant, i.e., (1 + 2r)-th time step. is more reasonable, more causal, and more real-time. For this reason, 3DConv might not be optimal for RSS.
across long-term
Transformer 30 is becoming the backbone of fundamental models in many computing fields 23 [6]. Compared with RNN, it overcomes the problem of parallel computing for handling sequential data; compared with convolution, it breaks the limitation of LRF by preserving the sequence-to- sequence encoding style. The key to Transformer's success lies self-attention (SA) mechanism. However, its problem is also with SA. By using parameterized connections with different weights, the sequence
is abstracted into the forms of {q: { (ke) and (vr)e= and then the temporal relation between any two primitives is obtained by densely calculating the pairwise

---

inner product along time dimension, i.e., [ /=.j== resulting the computational complexity of O(T2) for temporal relation modeling. This greedy manner is acceptable for data where only time-dimension is a concern, but for spatio-temporal tensors with large spatial ranges, e.g., radar data, the computational cost versus performance gain ratio may not be cost-effective. Moreover, since predictions are required only on current time step, computing resources should not be blindly allocated to each historical time step of input, which would lead redundancy.
The above analysis motivates this paper to redesign ppatio-tempora encoding module for RSS task, and teases out the following 5 design principles:
Utilizing parameterized rather than probabilistic connections to characterize the temporal relations of sequence like Transformers and RNNs do.
On the premise of making predictions at current time step, the module should emphasize the use of the current input frame, i.e., the non-context calculation in time.
The module should be able to handle temporal relations in parallel.
Considering the high-dimensional spatio-temporal characteristics of radar data, the module should ensure the efficient learning ability of long-term relationship and keep the parameters appropriately scaled.
Due to the coupling of device noise and environmental clutter, radar data would be non-smooth in time dimension. In this way. the module should consider the contribution of each time step differently during historical information aggregation.
To take into account the above principles, novel temporal learning paradigm for RSS is proposed as shown in Fig. (c)) i.e., data-driven temporal information aggregation with learned target-history relations. The careful designs are as follows:
The idea of modeling temporal relations with parameterized connections in RNNs and Trans- formers is preserved, i.e., temporal relation is explicitly treated as an intermediate embedding obtained by a parameterized layer/block (relation embeddings mentioned in $$ $$1.I).
The curren//aaget--histo this way, the complexity of ((T2) for temporal relations in SA is avoided (O(T) for TRAM), the use of the current frame is emphasized, and the convolution is non-context calculation time.
relation encoding mechanism is introduced (see TH-TRE in $$ 3.I. In
In order to achieve temporal relation encoding in the parallel way, temporal--elation-ineeee inceeption convolution is designed in TH-TRE (TRIC mentioned in $$ 3.1). which is also helpful to keep model scale constant with variable time length of input sequence.
learnable temporal pooling layer (see TRAP 3.2) designed to measure and reallocate the contribution degree of different target-history relation embeddings, thus the adverse effects caused by time non-smoothing of radar data quality can be alleviated in a data-driven way.
Temporal Relation Attentive Model (TRAM)
As illustrated in Fig. 2 the proposed TARSS-Net is based on CAED framework, which consists of basic encoder, TRAM, latent space encoder (LSE) and decoder. For each single-view radar input sequence, the basic encoder is used to generate high-level representations. The LSE is used to align and fuse the high-level semantic features of different views, which is further applied to the single-view decoder to improve its performance. Decoder receives inputs from TRAM and LSE, and finally produces segmentation results on RD and RA perspectives, respectively. As the key of temporal relation learning, TRAM will be detailed in this section, and remaining components can be found in Appendix C TRAM contains of two components: the target/current-hing -hittoriia temporal relation encoder (TH-TRE) and temporal relation-aware pooling (TRAP).
3.1 Target-Histon Temporal Relation Encoding (TH-TRE)
TH-TRE aims at capturing temporal relations of encoded target frame and its adjacent historical frame features. To achieve this goal, we design the remppoal--elation-inneppoi block to handle each hiittr-littoriaa feature pair, as shown in Fig. 3 Given the feature map sequence
convolution (TRIC)

---

TRAM
Hireical OTe TRINS
RD Enceder
RATCE
RD Predictor
Decoder
TRAM
AD Encader
Buvage
AD
Decader
Predictor
RA Enonder
Ms-D just TXE
Muli-View TARSS-Nc
Juhss Productio
Mult-niew ouder oipa segaoraze
Singosies croake (calao sepancc
Singonkev Ampont oargooed cntecdang
TC
Terporil conpresdoa
Maltissin boand quace umbhudinng
Figure 2: The illustration of multi-view TARSS-N The segmentation results for the radar data in RD and RA views, well referenced detection results the camera image presented for intuitive illustration.
(X,-7, X,-1, X,} [Cx(++])xfIx obtained from basic encoder (e.g., RA Encoder), the whole process of TH-TRE can be formalized Eq. ( C, (T+ 1), H, W are the numbers of channels/depth, time frames, range cells and angle cells, respectively.
TH-TRE ((X))==--)) = (TRIC (X,, (x.))=)))) @7 Max (X.)),
(1)
where TRIC (X,, (X.) (K2(K(()) & (X.)) 1-1 S
Where (.(+) C RC XWW, and K2(-) : : x CxHxxw, are 2D convo- lution layers, the operator @ and $7 denotes concatenation on depth and temporal dimension, respectively, Max is the 2D max-pooling operation with the spatial downsampling rate of 2. In this work, H H/2, W1 = W/2, H2 H/4, W2 = W/4.
It can be seen that (C, (-) drags input feature maps of different time frames into common feature space to ensure representation compatibility 34 and then K2(-)) takes enhanced feature maps of each compatible feature pair, i.e., IC₃ (X,) + (X.)). input learn the relations between target frame and its historical frames. In this way, the associations between target feature maps. X,, and each of its historical feature maps, (x, (X. is obtained. Finally the outputs from TRIC will be grouped together with (X,) to form the output relation embeddings, i.e., (X,-r, shown in Fig. 3 worth noting that convolution kernels in TH-TRE are shared, which helps it accept input with adjustable time length while keeping the number of parameters constant.
X.}
TH- Block
3.2 Temporal
TRAP
Relation-Aware Pooling (TRAP)
2B Canrllatios
Ohipoe Relation
Cravelaa Resed
The relation embeddings obtained by TH-TRE block are still in a temporal sequence form. While, for making final predictions on the target frame, the aggregation for generating summa- rized representation of entire input sequence is required. To this end, a learnable temporal pool- ing, i.e., TRAP, is proposed. In general, the TRAP block aims at perceiving the contribution degree of each historical frame for prediction task according to the target-history relations, and using these measurements of importance to aggregate the temporal information in each
Cany 2
Ceey.
Cass.
Conyr
Conyr
Conv:
Cars
Single-Vies
Exceder
Siegle Feaibr Seppenee
Figure 3: The illustration of TH-TRE.
5

---

Figure 4: The illustration of two forms of the proposed TRAP block: (a) Spatio-TRAL TRAP with Depth- Compression; (b) Depth-TRAP with Cpatii-Compeession.
single-view radar sequence. However, due to the computational pressure and parameter expansion caused by high-dimensional spatial-time tensors, it necessary to consider the trade-off of informa- tion utilization in each dimension before aggregation. In turn, two forms of TRAP are presented, i.e., main steps: compression
Spatio-TRAP and Depth-TRAN For each form of the TRAP, there (discard) and attentive pooling (aggregation).
Spatio-TRAP. Spatio-TRAP is performed on the entire spatial domain of input feature maps. There- fore, the importance of temporal relations will be estimated on the spatial space of relation embeddings.
As shown in Fig. (a). given the relation embeddings,
E ()) W2, the two i=t-r
main steps of Spatio- TRAP are as follows:
i. Septh-Compressionn First, all the relation embeddings are compressed by both global depth average- perrage-pooling and max-pooling. Then a temporal-shared 2D convolution block is used to generate the contribution map sequence, {M.) E (=x for following aggregation step. For each M, the process can be formalized as follows,
M, 1 (KST (Avg (X,) @ Max PC (X.))
Where Avg/ Vrg/Max"C denotes the global depth saaage//mx-pooliinng operation, (.) BaxH2xW5 + RI0xH2xW, and 1:T 116x + RRIH2xX2 are temporal-shared con- volution layers with kernel size of 3 and 1 x 1, respectively.
ii Opatio-Tempone Attentive Pooling. To modulate the input feature maps in spatio-temporal dimen- sion, a 2D softmax is performed on each map of (M.) Moreover, as deriving from the relation
) the contribution maps have a smaller spatial size than {X,) i.e. embeddings (f X i=
[H2, W2] [H, W]. Hence a spatial interpolation operation for expanding the size of (M.) required, so as to ensure the dimension consistency the following weighted summation and skip connection. Then the whole process of this step can be formalized as
ST
WST O
X
+ X,, where,
td
I=(-T
d=1
Intp (2DSoftmaxx MM),, (H/ H2, W/Wa))
S
HW/H,W
Where, XŞT and WST are enhanced target representation and spatio-temporal contribution weights of the i-th frame. Since the nearest interpolation Intp(., I) expands HW/ HW/H2W W2 times the spatial size of 2DSof tmax 2DSoftmax( (M), the values of W should be shrank to satisfy Σ E==,,=,,
1,,
ST
i, 1.
Depth-TRAP. Depth- Depth-TRAP is performed on the depth of input feature maps. During learning process, features expanded by the depth dimension are of great significance for unit-level classification (semantic segmentation), and they adequately represent semantic information. In this way, Depth- TRAP measures the importance of temporal relations semantic space of relation embeddings. Similar with the spatio form, Depth- TRAP also requires compression and aggregation steps (see Fig. (b)), which are as follows:
Comtiies-ommessiinn The input relation embeddings are first compressed in spatial domain by global average-pooling and max-pooling, then these two global pooled sequences are concatenated

---

and presented to sapporal-shared small MLP network, (*) R - R,, to obtain the spatio- compressed sequence, [x i This process can be formalized as follows,
gbC (Avg (Avg"(X,,)
(X.) + Max Max)(
(4)
Where Avg/Max²00 (.) denotes global average/max-pont pooling operation on spatial domain, with a kernel size of H2 22.
Attentive Pooling: This step calculates the contribution vector for each spatial- compressed temporal relation embedding. x;C, through depth dimension, which achieves the impor- tance mcasurement of temporal relations in semantic space. Then, with these contribution vectors, a weighted summation along with skip connection is conducted to aggregate each single-view encod- ing feature sequence, {X, Y -i=t-r- and therefore the enhanced tagees representation, X E Xx is obtained. The contribution vectors can be obtained as follows,
WDr Softmax
(VV
where,
Pi.d 0.1 sin (1/109t/22//)) if mod = 0: V, a Pr,d pr d=1² Pi,
(5)
C
Pi.d 0.1 cos (1/108((01)/2//0)) if d mod
Where (P)( RC + RC denotes the temporal-shared MLP, and pı,d is the depth-temporal position
encoding result following self-attention (30) for each element of {x([)== - Similar to Spatio-
TRAP, the contribution vectors scaled by softmax to jointly modulate the input feature maps in both depth and time dimensions. Then the scaled vectors, WDT (x++1))) are used to aggregate information from input feature maps as follows,
H.W
X
wPr O N
X
(6)
1=1,==1
Where the operator @ denotes Hadamard product, and X, RC the feature in each spatio- temporal position of input feature maps (X:) obtained from basic encoder. Finally, the output target representation, X,, enhanced by the TRAM will be presented to the corresponding decoder.
4 Experiments
4.1
Datasets and training setup
Three datasets have been used to valid the performance of TARSS-Net including: CARRADA. which contains multi-view annotated radar recordings (RAD tensors) for 4 categories of objects in driving scenarios under different weathers conditions; CARRADA-RAC [35], which is an improved version of CARRADA with calibrations RA view; KuRALs, which is real-measured dataset with refined annotation collected by a Ku-band surveillance radar and includes various typical targets. The training setup for TARSS-Net and other compared SoTA networks strictly follows the consistent configuration, including hardware platform, hyper-paramen settings and evaluation metrics, and the number of input frames for TARSS-NetS is by default (7 4). See Appendix D more details.
4.2 Comparisons with State-of-The-Arr Methods
The most widely used RSS dataset, CARRADA, is used to comprehensively compare the performance of -TRSS-NNe with existing RSS methods. MARRADA-RAS and KuRALS dataset are utilized to explore the generalization and stability of MARSS-Net on which only representative methods are compared, i.e., TMVA-Net (temporal- based model) and PKCIn-Net (current SoTA). TARSS-Net and TARSS-Net D denote the TARSS-NO with Spatio-TRA and Depth- TRAP, respectively.
i. CARRADA. The overall results on the test subset are listed in Table T Generally, the models that explicitly consider the temporal dynamies in radar signals, e.g. TM
21. can obviously
achieve better performance than other convolution-bass models 200 28
4
6
By introducing SA
7

---

mechanism in Transformers [30 18 into convolution frameworks, TransRadar [5], T-RODNet [3 (the RD scores are not provided in the literature) and TransRSS [66] (the code is not public in the literature) further enhance the representation ability in spatial dimension, thus pushing the performance of RSS to a new height. However, due to the explosion of parameter scale caused by the high-dimensional radar tensors, none of them use SA in temporal domain. Compared with the existing RSS methods using temporal information, TARSS-Nets with the well-designed temporal learning paradigm exactly follows the 5 design principles mentioned in $ $2 and achieves the best overall performance (average metrics for both views) with reasonable parameter scales. Unfortunately, its RA performance is not so prominent, as the RA data does not change much over time or poor angular resolution, resulting in less obvious temporal information that can be exploited Specially, PKCIn-Net [35] does not belong to the category of temporal information modeling, but its better performance the RA view shows that is helpful to introduce the classical radar signal processing theory into RSS models, which provides direction for the future improvement of TARSS-Nets.
ii CARRADA-RAC. Models listed in Table T fail to aquire the ideal performance on the RA view. One main reason is the poor angular resolution of common low-cost FMCW radars. CARRADA- RAC (35) has calibrated the RA annotations, which could give a more realistic performance compari- son for different models. As shown in Table 2 the global performance are improved after annotation refinement, especially in RA view and TARSS-ne still achieves the performance of SoTA.
This dataset is different from CARRADA in terms of radar systems and detection scenarios, which could complement testing the generalization and stability of models. Experiments in Table 3 show that TARSS-NG cloud maintain its superior performance in different application scenarios. The model marked with s has been modified to a single-view version as needed.
iii. KuRALS.
Table 1: Comparisons with SoTA RSS networks.
Table 2: Performance on CARRADA- RACC
Method
#Parum.
mloU
RD-View (%)
miDice
mlol
RA-View (%)
mDDice
View
Method
#Param.
mloU
mDice
FCN(20
134.3M
66.3
40.9
TMVA-Net
5.6M
59.7%
69.9%
U-Net
17.3M
55.4
68.0
32.8
38.2
RD
PKCIn-Net
6.3M
60.6%
72.4%
DeepLaby3+
2
59.3M
61.6
TARSS-Net.S
6.2M
62.5%
74.3%
RSS-Ne((L4)
10.1 M
32.1
36.9
32.1
37.8
TARSS-NeL
6.3M
62.8%
74.6%
RAMP-CNN
6
106.4 M
56.6
68.5
27.9
30.5
TMVA-Net
5.6M
46.6%
57.9%
MV-Net
2
29.0
32.8
26.8
28.5
PKCIn-Net
6.3M
47.3%
58.7%
211
21
5
M
56.1
53.5
65.3
68.0
37.1
37.7
46.2
44.8
RA
TARSS-Net
6.2M
45.8%
56.1%
TransKadar's
49M
57.2
09.1
3933
29.3
TARSS-Net_
6.3M
47.4%
58.7%
L-RODNe(
162.0M
43.5
53.6
TMVA-Net
5.6M
53.2%
63.9%
TransRSS 36
60.4
73.0
43.0
53,8
Global
PKCIn-Net
6.3M
54.0%
65.6%
35
6.3M
60.7
72.6
43.1
53.7
TARSS-Net
6.2M
54.1%
65.2%
TARSS-Net_S 8
M
62.1
73.8
41.6
51.2
NeRSS-Net_D
6.3M
55.1%
66.7%
TARSS-Net
6.3 M
63.4
75.2
41.4
51.3
4.3 Ablation Study
Three sets of ablation experiments are conducted to verify the effectiveness of proposed TRAM module and its two independent blocks, TH-TRE and TRAP. Depth-TRAP is used as default.
Ablation Experiments on TRAM. As control experiments, two baseline models designed by incorporation different mechanisms of tem- poral information utilization to TARSS-Net: i)
Table 3: Performance comparison on KuRALS.
RD View
Method
#Param.
mloU
mDice
Baseline-A: replacing TRAM from TARSS- with single temporal global average/max-pone -pooling (GAP/GMP) layer; ii) Baseline-B: B: replacing TRAM from TARSS-Net with temporal aggregation lay- ers designed in TMVANet (21). which consists of stacked 3DConvs. Results show in Table 4 Baseline-B B achieves better performance on both views than Baseline-A, which indicates that parameterized learnable aggregation has obvious advan- tages over parameter-free (max avg) aggregation. While by using the proposed TRAM, TARSS-Net can obviously exceed 3DConv-based Baseline-B on almost every evaluation metric, which fully demonstrates the effectiveness of the proposed temporal-aware learning paradigm for RSS networks.
FCN
134.3M
50.4%
59.4%
U-Net
17.3M
52.4%
60.1%
DeepLaby3+
59.3M
52.6%
61.8%
TMVA-Nett"
1.2M
52.9%
63.1%
PKCIn-Net
1.2M
56.7%
65.9%
TARSS-NeL.
1.2M
53.2%
63.8%
TARSS-Net D
1.3M
58.4%
67.1 %
8

---

The Effectiveness of TH-TRE. TH-TRE aims to capture temporal relations of target frame and its historical frames. To verify its effectiveness, two more contrast models are presented: Baseline-A w/ GAP & TH-TRE: adding TH-TRE block to Baseline-A w/ GAP, in which a single temporal GAP layer is used to aggregate the temporal relation embeddings obtained from TH-TRE; ii) TARSS-Ne w/o TH-TRE: TARSS-NG without explicitly learning temporal relations of the target frame and its historical frames. Here, the TRIC block is replaced with a regular 3DConv block, which contains two convolution layers with the same kernel size of 1 X 3 X 3 and stride of (1, 2, 2). As shown in Table 5 the addition of TH-TRE block is beneficial to improve RSS performance both Baseline-A w/ GAP and TARSS-Net w/o TH-TRE. This block realizes current/target -hittrr relation encoding under the premise of maintaining temporal order. In this way, TH- TRE emphasizes the use current frame and achieve more computationally efficient and powerful temporal encoding capacity than simple temporal compression mechanism such as GAP and 3D convolution.
Table 4: Ablation experimental results on TRAM.
Method
RD- View (%)
RA-View (%)
Global (%)
Prec.
Recall
mlot.
milice
Frec.
Recall
mlol
milice
Prec.
Recall
mlol
miDice
Baseline-A A w/ GAP
61.8
76.3
55.4
67.8
43.5
47.1
36.4
44.7
52.7
61.7
45.9
56.3
Baseline-A GMP
68.2
50.8
62.1
42.9
42.7
55.6
43.5
52.4
Baseline-B B
63.6
74.8
56.1
68.0
44.2
51.6
37.7
46.2
53.9
63.2
46.9
57.1
NeRSS.Net
80.6
752
50.0
51.3
65.3
63.3
Table 5: Ablation experimental results on TH-TRE
Method
RD-View (%)
RA-View (%)
Global (%)
Prec.
Recall
mloU
mDDice
Prec.
Recall
mioU
mDicee
Prec.
Recall
miol
mDiiee
Haseline-A W/ GAP
61.8
76.3
55.4
6188
43.5
47.1
36.4
44.7
5277
61.7
45.9
56.3
Basclinc-A & TH-TRE
68.9
79.3
60.6
72.3
50.9
50.2
40.4
49.7
59.9
64.8
50.5
TARSS-Net w/o TH-TRE
69.9
79.2
61.6
73.7
53.2
50.1
40.4
49.8
61.6
64.7
51.0
61.8
TARSS-Net TH-TRE
70.9
80.6
63.
75.2
56.1
50.0
41.4
$1.3
63.5
63.3
52.4
63.3
Table 6: The effects of Depth/Spatio TRAP block.
RD>Vieww (%)
RA-View (%)
Global- Viel--iiewww (%)
Method
Prec.
Recall
mlol
mDiice
Prec.
Recall
mloU
mDiiee
Prec.
Recall
mlol
mDiice
Baseline-A_
61.8
16.3
55.4
67.8
43.5
47.1
44.7
52.7
61.7
45.9
36.3
Baseline-A_S
69.5
78.5
61.0
73.3
50.8
49.4
38.8
47.4
60.2
64.0
49.9
60.4
Buseline-A_D
69.9
79.2
61.6
73.7
53.2
50.1
40.4
49.8
61.6
64.7
51.0
61.8
TARSS-Net_ G
68.9
79.3
60.6
72.3
50,9
50.2
40.4
49.7
59.9
64.8
50.5
61.0
TARSS-Net S
70.4
79.1
62.1
73.8
53.6
50.6
41.6
51.2
62.0
64.9
51.9
62.5
TARSS-Net D
70.9
80.6
63.4
75.2
56.1
50.0
41.4
51.3
63.5
65.3
52.4
63.3
The Effectiveness of TRAP. TRAP aims to measure and allocate the contribution degree of different target-history relation embeddings before aggregation. Several models are eval- uated to valid its effectiveness: (i) Baseline-A_Gi Baseline-A with GAP; (ii) Baseline- A_S/D: Baseline-A with Spatio/Depth-TRAL block from TARSS-Net with GAP; (iv) TARSS-Net S/D: TARSS-Net with Spatio/Depth- TRAP block. The results listed in Table 6 show that either form of TRAP block can aggressively improve the performance of both Baseline-A model and TeRSS-Net. This illustrates that different history frame has unequal closeness with target frame, and TRAP would be an efficiently way for reassign- ing weights to temporal relations aggregation, which effectively alleviates the negative influ- ence RSS caused by temporal unsmoothness of low-quality radar data. In addition, Depth- TRAP- models always obtain better results than Seatio-TRAP based models. Such perfor- mance gap may caused by the aggressive com- pression of the semantic information (reflected in the channel dimension of features) in Spatio- TRAP block. As mentioned in $$ 3.2 Depth- TRAP pays more attention to the semantic fea- tures obtained from the TH-TRE block, and such consideration makes the network generate more informative representations than Spatio-
block; (iii) TARSS-NeI_G replacing TRAP
Net Performance variation under diffirenn time lengths
3.5
4,007
3302
(366)
676
07000
(3056
6
01771
0150
(335
1700
0e5
0/02
0/423
10
Figure 5: TARSS-Net performance (RD view) vari- ation on CARRADA using input sequence with different numbers of frames.

---

TRAP. Therefore, we argue that maintaining more semantic information might be an effective way to further improve Spatio- TRAP, which is worth of investigating in the future work.
4.4 Effect of input time length on TARSS-Net performance
The RD performance of TARSS-Net increases with time length of inputs grows and achieves the optimum at frames, i..., 5, as presented in Fig. 5 With the careful design of temporal modeling paradigm, TARSS-Net can handle arbitrarily long sequence relations without increasing parameter scale, which is significant to find the optimal duration of temporal dependencies not only for RSS tasks but also for all temporal relation modeling methods.
4.5 Real-time performance comparison
In addition to model scale, the computational complexity and inference speed of the RSS methods are also important for deploying applications, which is measured by multipyy-accumulate operations (MACs) and frames per second (FPS), respectively. All the real-time performance shown in this section are obtained
single RTX 3090 GPU.
Among existing RSS models, TMVA-Net is the SoTA one for multi-view temporal relation learning. As far as knowledge, none of existing methods applied Transformer SA on the time dimension in RSS task because of the exploding number parameters. In order to deeply explore the real-time performance of different temporal relationship modeling methods, introduce vision Transformer (ViT) into the basic architecture of TARSS-Net, which forms a ViT-based baseline method, called Vit-based-Net. The left part of Table 7 compares the real-time performance of TARSS-Ne with TMVA-Net and Vit-based-Net with multi-viww inputs. All methods take consecutive time frames as input. It can be seen that directly introducing Transformer temporal relationship modeling of radar data is inefficient, which would consume more computation source but obtaining lower RSS performance. TARSS-Net consumes more computation and inference time than TMVA-Net in temporal relation learning, but achieves better RSS accuracy. In addition, the inference speed of 23 FPS in TARSS-Net D is also sufficient for real-time applications.
Table 7: Real-time performance (MV: milti-view; SV: single-view).
Method
Inputs
Params
MACs
FPS
mloU (%)
mDice
Inputs
Params (M)
RD-View
RA-View
(M)
(G)
(%)
MACs(G)
MACs(G) FPS
IMVA-Net
MV
7.2
119.5
66
46.9
57.1
SV
1.2
11.3
250
36.6
250
Vit-based-Net
MV
27
449
12
38.1
44.5
SV
3.6
1.8
59
7.0
55
NeLLS
MV
6.2
197.6
35
51.9
62.5
SV
1.2
13.3
181
40.4
143
TARSS-Net_D
MV
6.3
175.4
52.4
63.3
SV
1.2
13.3
111
40.4
112
The right part of Table 7 compares real-time performance of several temporal relationship learning models in single view. That is, the results for RD and RA views measured separately by the corresponding single-view network versions. In terms of real-time performance, TMVA-Net still has the fastest inference speed. But Vit-based-Net achieves the worst inference speed with the most number of parameters. However, it is not to be overlooked that TRASS-NG -ett still leads the pack in terms of RSS performance with competitive real-time performance.
More supplementary experiments are presented in Appendix t including ablatioin experiment of AD encoding branch, class- wise performance comparisons, features visualization of core process in TARSS-NG and some example visualization of segmentation results.
Conclusions
This work focuses on exploiting temporal information in radar signals to enhance the representation capacity of multi-view RSS network. Firstly, the advantages and disadvantages of existing temporal modeling methods in RSS domain were deeply discussed, and on this basis, the design principles of RSS spatio-temporal encoding methods were introduced. Based on the principles, flexible tewporal-aware learning module. TRAM, and TARSS-NE based on TRAM is proposed, which follows the novle temporal learning paradigm, i.e. data-driven temporal information aggregation with learned target-history relations. Experiments fully verifies the superiority of TARSS-N through SoTA methods comparison on three datasets, ablation experiments, performance under variation input time length, well as its real-time performance.
10

---

Acknowledgements
This work is supported by National Natural Science Foundation of China (No. 62206258).
References
[1] Daniel Brodeski, Igal Bilik, and Raja Giryes. Deep radar detector. In 2019 IEEE Radar Conference (RadarConf), pages 1-6, 2019.
[2] Liang- Chighh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs/1706.05587 2017.
[3] Liang- Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision ECCV 2018, pages 833-851, Cham, Springer International Publishing.
2.18.
[4] K. Cho, Van Merrienboer, C. Gulcehre, D. Bahdanau, Bougares, H. Schwenk, and Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. Computer Science, 2014.
[5] Yahia Dalbah, Jean Lahoud, and Hisham Cholakkal. Transradar: adaptive-directionn transformer for real-time multi-view radar semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 353-362, 2024.
[6] Xiangyu Cao, Guanbin Xing, Sumit Roy, and Hui Liu. Ramp-cun: A novel neural network for enhanced automotive radar object recognition. IEEE Sensors Journal, 21(4):5119- 2021.
[7] A. Graves. Generating sequences with recurrent neural networks. Computer Science, 2013.
[8] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE International Conference on Acousticx, Speech and Signal Processing. pages 6645-66499 2013.
[9] Daniel Gusland, Sigmund Rolfsjord, and Børge Torvik. Deep temporal detection machine learning approach multiple-dwell target detection. In 2020 IEEE International Radar Conference (RADAR). pages 203-207, 2020.
[10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. IEEE Transactions Pattern Analysis and Machine Intelligence, 42(2):386 397, 2020.
[11] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 37(9):100-19116 2015.
[12] S. Hochreiter and Schmidhuber. Long short-term memory. Neurol Computation, 9()):1735-1780, 1997.
[13] Tiezhen Jiang, Long Zhuang, Qi An, Jianhua Wang, Kai Xiao, and Anqi Wang. T-rodnet: Transformer for vehicular millimeter-wav radar object detection. IEEE Transactions on Instrunentation and Measurement, 72:1-12, 2022.
[14] Prannay Kaul, Daniele de Martini, Matthew Gadd, and Paul Newman. Rss-net: Weakly-supervised multi-class semantic segmentation with fmew radar. 2020 IEEE Intelligent Vehicles Symposium (IV), pages 431-436, 2020.
[15] D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015.
[16] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yes Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3992-4003, 2023.
[17] Tsung- Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR). July 2017.
[18] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
11

---

[19] Zhang Liwen. Pan Jian, Zhang Youcheng, Chen Yuanpei. Ma Zhe. Huang Xuhui, and Sun Kewu. Capturing temppraa--dependence radar echo for spatial-lemporal sparse target detection. Journal of Radars, 12(R22228)3356, 2023.
[20] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[21] Arthur Ouaknine, Alasdair Newson, Patrick Pérez, Florence Tupin, and Julien Rebut. Multi-view radar semantic segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15671-15680, Octoher 2021.
[22] Arthur Ouaknine, Alasdair Newson, Julien Rebut, Florence Tupin, and Patrick Pérez. Carrada dataset: Cam- era and automotive radar with range- angle- doppler annotations. 2020 25th International Conference on Pattern Recognition (ICPR), pages 5068 -0788,55757707 2021.
[23] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730 27744. Curran Associates, Inc., 2022.
[24] Qizhe Qu, Weijian Liu, Jiaxin Wang, Binbin Li, Ningbo Liu, and Yong-Liang Wang. Enhanced cnn-based small target detection in sea clutter with controllable false alarm. IEEE Sensors Journal, 23(9):10193- 10205, 2023.
[25] L. Rabiner and B. Juang. An introduction to hidden markov models. IEEE ASSP Magazine, 3(1):4116, 1986.
[26] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
[27] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137-111 2017.
[28] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Nassir Navab, Joachim Homegger, William M. Wells, and Alejandro Frangi, editors, Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, pages 234-241. Cham, 2015. Springer International Publishing.
[29] Tran, Lubomir Bourdev, Rob Fergus. Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 4489 4497, 2015.
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems. volume 30. Curran Associates, Inc., 2017.
[31] Hao Wan, Xiaoqing Tian, Jing Liang, and Xiaofeng Shen. Sequence-feature detection of small targets in sea clutter based on bi-lstm. IEEF Transactions Geoscience and Remote Sensing, 60: 1-11-1 2022.
[32] Jingang Wang and Songbin Li. Maritime radar target detection in sea clutter based on enn with dual- perspective attention. IEEE Geoscience and Remote Sensing Letters, 20:1-5, 2023.
[33] Yizhou Wang, Zhongyu Jiang, Yudong Li, Jenq-Neng Hwang, Guanbin Xing, and Hui Liu. Rodnet: real-time radar object detection network cross-supervised by camera-radar fused object 3d localization. IEEE Journal of Selected Topics in Signal Processing, 15(4):954-967. 2021.
[34] Jiarui Xu, Yue Cao, Zheng Zhang, and Han Hu. Spatial-temporal relation networks for multi-object tracking. In 2019 JEFF/CVF International Conference on Computer Vision (ICCV), pages 3987-3997, 2019.
[35] Liwen Zhang, Xinyan Zhang. Youcheng Zhang, Yufei Guo, Yuanpei Chen, Xuhui Huang. and Zhe Ma. Peakconv: Learning peak receptive field for radar semantic segmentation. In 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 17577-17586, 2023.
12

---

[36] Hao Zou. Zhen Xie, Jiarong Ou, and Yutao Gao. Transrss: Transformer-based radar semantic segmentation.
In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 6965-6972, 2023.
13

---

Appendix
Following parts are introduced in this appendix: (i) the input streams for our network, i.e., radar signal processing for mueti-view range-angle-Doppler (RAD) tensors; (ii) more detailed contrastive analysis of present deep learning-base based sequential models, e.g., Self-Attention [3], RNN family g 7 and our temporal-awand aware learning mechanism; (iii) additional descriptions of our TARSS-Nel and TRAM module; (iv) more detailed description of datasets and training setup; (v) additional experiments including ablation of AD Encoding branch, feature visualization and some example visualization of segmentation results; (vi) limitations and social impacts.
A From Radar Signals to Multi- View Representations
As mentioned in $. most CAED-based radar destctionnssemmenttionn methods use FFTs signal processing front-end. Despite the loss of fine-grained temporal information, FFT enables pure time domain radar echoes to be expressed in spatial (range and angle) and Doppler. At the same time, it can provide more intuitive and structured input to the model. Therefore, we also use such classic signal processing to aquire the multi-view RAD tensors.
Considering one-frame radar signals received from an FMCW radar, is composed of multiple chirps from multiple antennas and can be denoted as [Chirp, (5) No.NA Ji=1,==1 where ND and NA denote the numbers of chirps and antennas, respectively. Then Range- FFT, Fx((), is conducted on each chirp,
Chirp, to obtain the DFT (Discrete FT) results, then we can get DFT tensor follows:
(Fr(Chirpp) (3) | No.NA (M 1 E NRNNNNNN
(S1)
Where M denotes the Range-DFT matrix for j-th antenna chirps, Nn is the sampling number of each chirp. Please note that we only consider the real part of FFT results for simplification. Then for (-), is conducted to get the Doppler-DFT matrix as follows: each row of M the Doppler-FFT, F-(.),
(FD(M [k, :))= M E NAXND
(S2)
Then can group these DFT matrixes of all antennas to form the second DFT tensor, {M MAD. Finally, the Angle-FFT, F-),,, is performed on MID along the antenna dimension get final RAD tensor:
NA
RD J=1
[Fa k, k))) Nn. No.No MRAD E R-NaXNNXNA
(S3)
In this work, {NR. NA. ND) (256, 256, 64). Obviously, even single frame of MRAD is also quite dense and bulky for deep models. Further compression is needed to obtain affordable input streams in different frequency domains. Therefore a 2D-based multi view compressing method is adopted in (21), i.e, averaging over different frequency domains. Taking angle frequency domain as an example, 3D RAD tensor would be compressed as 2D RD view representation follows:
NA
RRDli, k] = 10 log
Σ
M
(S4)
NA j=1
Using such processing, the multi-view representations of our models, (XRD. XAD, XRA} of one- frame radar tensor can be obtained. And the input scale is agressively reduced from 256 256 x 64 to 256 x 64 + 256 64 + 256 x 256.
B Contrastive Analysis of TRAM and Other Sequential Models
This section complements the discussion about Temporal Information Utilization in $
For present, one of the important problems is insufficient use of temporal information or clues in radar data. Present works [ 14 6 21 33 mainly focused on providing a complete solution to radar scene understanding tasks, while spatial-temport information is relatively simply introduced by some proven techniques in other fields, e.g., 3D convolution or its variant deformable convolution. That is
14

---

Using histurica information rnhance target frame
Relario Eickaseed Reye Reperinnationn
represrntation and prediction
Capturing margee hinrry relntions)
Azgrouiing whole sequence with weighted relutious
Aggrgattion
W
W
Ws
Temporal Impartunce
X
Hisorical Frames
Tegsst
Figure S1: The basic learning paradigm (left) and implementation of proposed TRAM (right). TRE in the yellow boxes denote TH-TRE; Temporal Relation Importance Measurement relates to the attention weights calculation in TRAP block, and Relation Aggregation relates to weighted summation and skip connection in TRAP block.
say, temporal modeling only plays a limited role in these works. However, in this work, dedicate to this temporal information utilizing problem, and attempt to define a general learning paradigm: capturing temporal relations first and then aggregating them by task-driven manner. These relations are extracted from target frame and its historical neighbors in an explicit way, then Icarnable relation aggregator would estimate the importance of those relations and group them adaptively into single representation according to the task. It is different from previous works which merely rely on convolutions and learning-fire aggregating operators, e.g., mhax/avg-pooling
As mentioned in $ the existing temporal modeling methods can be summarized into two categories: i) Causal temporal relation modeling, represented by HMM and RNN family; ii) Parallelized se- quence representation modeling, represented by 3DConv and Transformer (SA) family. Despite the advantages of these methods, directly using RNN and SA would introduce other problems. For RNNs, the dependencies between elements are captured by recursive dependency calculation manner. That to say, produce an output, O, at f-th frame, the former one O,-1 should be generated first. Such recursive dependency encoding mechanism could not fully enjoy the convenience of parallel computing. For SA, the dependencies between elements are captured across each pair of elements in the input sequence. However, extracting all context relations could cause redundancy. Since our task make prediction on target frame, reasonable pay more attention target-history pairs instead of each pair. Moreover, SA is more friendly to MLP-based frameworks, but for convolutional networks, its multi-scale spatial encoding capacity would be restricted some extent since there is trade-off between information compression and parameter scale when flattening the spatio-structured feature maps into vector. Comparatively speaking, the proposed TRAM module well overcomes above shortcomings. It can capture the temporal relationship of variant input time length without increasing scale of network, and at the same time retain the advantage parallel computing.
To intuitively show our basic idea and parallel computation of proposed TRAM, we drew Fig. SI One implication can be seen in the figure, that the importance measurement can be regarded soft-selection for higeet-hstoryy relations. It actually drives the model to focus on the useful part of history information. That say, if one can decide which historical frames contribute most to target frame representation and prediction, it would be easier to design model with longer temporal dependence and less learnable parameters, since history information with less importance could be ignored directly. And we argue that filterin useful historical information in advance is also worth investigate in the future, so that the model can "see more history", while keeping a reasonable parameter scale.
15

---

TRAM
Range
Historical time Srame
Enender
RD
RD
TRAM
Deppler
AD Encuder
Range
Range
RA Decoder
RA Predictor
TC
time frame
RA Encoder
RATCE
TRAM
RA-AD join TAE
Range Angle Duppler
Multi-View TARSS-Net
Joint Prediction
Malsiviee imput sequences
Temporal (TC)
Singleview mncdia
Singl-view temporal- compreseed embeddinn (TCE)
Multi view laten: space embezading
feature soquenee
RD-AD-RA-AD joint tampera-ewoore embedding (TAE)
Figure S2: The simplified workflow of TARSS-Net. The segmentation results for the radar data in RD and RA views, as well referenced detection results the camera image are presented for intuitive illustration. ASPP module is omitted for simplicity.
C Additional Descriptions of TARSS-Net
C.1 Detailed description of TARSS-Net framework
As illustrated in Fig. S2 the proposed TARSS-Net is based CAED framework. In addition to TRAM, some core components are introduced as follows.
Basic encoder. For each single-view radar input sequence, the basic encoder is used to generate high- level representations, and multi-view radar sequence can handled by performing these encoders in parallel. All three single-view encoders are designed with the same structure for simplicity. Each encoder is mainly composed two 3D convolution blocks with kernel size of x x 3, which shares convolution kernel learning of each frame the time dimension, between all historical and target frames. worth noting that this different from using 2D convolutions, which do not enable parameter sharing and feature interaction in dimensions outside the 2D plane. To further compress the spatial size of feature maps, each convolution block follows spatial max-pooling layer.
as realize feature interaction
Latent Space Encoder (LSE). The basic encoder and TRAM are performed on each single-view of the radar sequence. However, one problem in multi culti-persspectvee learning structure is how to extract the rich knowledge from different information flows, and use this knowledge to improve the performance of the model each local sub-task or global task. For this purpose, multi-view shared learner, LSS added to our TARSS-Net. The inputs of LSE are cemporal-comppessed embeddings (TCEs) obtained by temporal compression (TC) module, which consisted with a global temporal average-poodl poraage-pooling layer and 2D convolution layer with 1 X kernels. Two shared 1 x 2D convolution layers are stacked to form the LSE, which will project the TCEs into common latent space to get uniform embedding. Since multi-view inputs are used, the 1 x 2D convolution layers are chosen instead of the fully connected layers for reducing the model parameters.
Decoder. After encoder, the features obtained from TRAM and LSE could comprehensively consider single-view temporal relationship and multi-view relationship alignment, which will be fed into decoder. Only two branches for RD and RA views are designed decoder, which is enough to complete target location. By the way, this is also the reason why existing RSS datasets is not labeled on AD view with groundtruth. Each decoders consists the combination of 2D deconvolutions and convolutions in (21)
16

---

TRAM
TARSS-Net
RD Lupe aequarc
Encoder
ASPP
RD Encoding Branch
XD corper prodiction
Decoder
AD regal
AD Encoding
Branch
Decoder
peodiction
RA Encoding Branch
sequeace
Tenporal ED Cour
Oimoiaation
Encoding Fire
Tempoed
Lataut Rpace Lncoding
Figure S3: The overall framework of TARSS-Net, details of the encoding branch for RD-view is given an example.
In general, Fig. S3 gives an intuitive illustration of the overall structure of our TARSS-Net. The encoding branch would take multi-view radar tensor (i.e., RD, AD, and RA) streams within several time frames as input, and generate the unified representation of these streams in a common latent space. Thus the encoding branch used learn representations and the decoding branch is used to interprete those representations. Besides each basic encoder, the proposed TRAM and ASPP modules aim to enhance the representation by capturing and measuring harget-history relations and injecting multi-scale spatial information from each single view. Using these enhanced representations as input, the decoder would make cell-wise predictions on RD and RA views, respectively.
C.2 More in-depth discussion of TH-TRE block
The TRIC (temporal cenppooa-rellttoon relation -nneeption convolution) block proposed in TH-TRE ($ 3.1 can also be implemented with the 3D convolution. Firstly, let's recall the introduced implementation of TH-TRE:
TH-TRE (Xx))==<>7) [TRIC (X,, (x.)=)) @ Max (X.)).
(S5)
where TRIC (X,, (X.) (Ks (x))) K
(X.))
Where (-) and K.).)) are 2D convolution layers. The 3D-based implementation TRIC can be conducted as follow:
TRIC (Xt: (x.) K2 (Ix; @ 1
(S6)
where -x;
= Ki ((x,)i=t-7)-
Where (.)()) DCx
Hx and (3)()) 22CxxxHixW
RCxxX H2 are two 3D convolution blocks with a kernel size of in temporal dimension. Thus, the kernels temporal-shared for each convolution layer. Then if 2D conv blocks of the same layer in 55 (the original) are shared, TRIC would be equivalent to its 3D version (TRIC*). In this case, the numbers of parameters of TRIC and TRIC" are the same, i.e., (CxIxxx3x))+(CXXXXXX
However, when we review the present implementations of TRIC and TRIC", we realize that they are all in stacked form, i.e., K2/AS IC; is always nested in K2//K5, It means that, the calculation of K K2/KC; this end, a more parallel and elegant implementation of TRIC" comes up, and Fig. S4 intuitively describes this implementation. The key idea of this implementation temporal cross reorganization for target-history feature pairs and shhped 3D based relation encoding with 2 x 3 x 3 kernels and time stride of 2. In this case. the
has to wait for the outcome KC, //KK,
17

---

20 Conv and Skip Conncction
Temporal Concatenation
Inpat Single-View Feature Map Sequence
Output Relation Embeddiing
3D Conv with
lepus Targst Faaure
Till Til-Rotation Embedding
TRIC with 2X3X3 kernels
Target-Hisstory Feature Pair
Fasturo Map
Eubeddings
Figure S4: A more parallel implementation of TRIC".
parameter scale keeping the with present TRIC" with x 3 3 kernels, while enjoying more parallel structure and less computation since there is depth-concatenation and computation of the two Conv blocks are decoupled. More experiments and analysis will be conducted in the future work about these implementations.
D Datasets and training setup
D.1 Datasets
Three datasets have been used to valid the performance of TARSS-Net, including CARRADA [22] CARRADA-RAC and KuRALs.
CARRADA is a camera-radat synchronised dataset with large scale, which contains multi-view annotated radar recordings (RAD tensors) collected from low-cost FMCW radar in various scenarios under different weathers conditions. There are categories of objects: pedestrian, cyclist, vehicle and background; and multiple same/different types of moving objects could appear in single time frame. The dimensions RAD tensor are 256, 256 and 64, respectively. For fair comparisons, RSS-Net and RAMP-CNN were modified as the multi-view version, and all the SoTA methods including our TARSS-Nets were trained with the same setup.
is an improved version of CARRADA. As mentioned in $$ 4.1 CARRADA is camera-radar synchronised dataset. RD and RA annotation of were generated
CARRADA-RAC
semi-automatic way [22]]. in which the unreliable depth estimation of camera image and low angle resolution of FMCW radar reduce the labeling accuracy, especially in RA view. CARRADA-RAN further improves
CARRADA by correcting its RA annotation accurate 35 The training, validation and test subsets
were split as 21
KuRALS large-scale self-collected real dataset with refined annotation. is produced by Ku-hand continuous wave radar and includes various typical targets such UAVs, airplanes, ships, and cars, which could support several radar sensing tasks including RSS, target tracking and so on. There only RD tensors KuRALS, so TMVA-Net, PKCIn-Net well TARSS-Net shown in Table 3 are modified to single view version. The complete dataset is forthcoming.
D.2 Training Setup
The training setup for our proposed models and other compared SoTA networks is strictly consistent follows:
Input form:
The raw 3D RAD tensor of each time frame was first compressed as 2D RA, AD and RD views with the input sizes of 256 256, 256 x 256 and 256 64. For the networks without explicitly considering the temporal information, e.g., RSS-Net [14] and MV-Net [2T]. 2 historical frames (i.e.,
2) were added to form the input sequence. Besides, for the temporal information-base networks, e.g., RAMP-CNN (). TMVA-Net (21), and our TARSS-Net, more historical frames were used:
8 for RAMP-CN CNN, and for TMVA-Net and TARSS-Net.
18

---

Running platform:
The experimental platform of this work is RTX 3090 GPU with 24G memory. Both the single-view version and the multi-view version of TARSS-Net can be trained and tested on single 3090 GPU.
Hyper-parameters:: All the models were trained with Adam optimizer [15] using the default setting of hyper-parameters, 31 = 0.9, 32 = 0.999 and e = le 8. The initial learning rate was 1e 4, which decayed exponentially with the rate of 0.9 every 20 epochs. The training epochs and mini-batch size were 300 and 6, respectively.
For comprehensive comparisons, evaluated methods using cell-wise precision/recall, Intersection over Union (loU), and Dice score. As the principle metrics, mean IoU (mloU) and mean Dice (mDice) are caleulated by category averaging. These two metrics used to compare the performance of different methods on three validation datasets, as shown in Table [ Table 2 and Table 3 In ablation experiments, shown in Table Table Table [6] precision and recall are also presented to comprehensively measure the effect of each design in TÄRSS-Net All the reported results were obtained from the test subset.
Evaluation metrics:
E
Additional experiments
E.1
Ablation experiment of AD encodingg branch
Since prediction results are generated on RD and RA views, AD view is actually embedded in the decoding branch. To further verify the effect of AD view in the multi-view R$S framework, we conduct an additional experiment on AD encoding branch (AD Enc.), both Baseline-B (replacing TRAM from TARSS-Net with emporal aggregation layers designed TMVANet) and TARSS- Net_De (TRASS-Net with Depth-TRRP) were used for the experiment. Results in Table clearly show that models can achieve better performance by using additional AD view. This performance improvements mainly come from the information compensation, i.e., AD view compensates RD with angle frequency response in angle domain and compensates RA with Doppler information. Such results also reveal that, taking full advantage of the information in the radar signals is indeed helpful for radar scene understanding. Hence from the perspective of information utilization, our method propose pathway to joint analysis of multiple spatio-temporal domains of radar data.
Table S1: Ablation AD Encoding Branch
RD-View (%)
RA- (%)
Method
mloU
mDice
mloU
mDiice
Bascline-B B w/o AD Enc.
52.8
64.3
35.5
42.4
Baseline-B w/ AD Enc.
56.1
68.0
37.7
46.2
TARSS-NeU w/o AD Enc.
60.1
71.9
39.8
49.2
TARSS-Net_D w/ AD Enc.
63.4
75.2
41.4
51.3
E.2 Comparison of class-wise performances
There are categories of objects in CARRADA dataset, including: pedestrian, cyclist, vehicle and background. The intuitive comparison of class-wins performances is shown in Fig. [S5] TARSS-Net D has the best overall RSS performance for most types of objects, especially for cyclists, while TARSS- Net_S performs better in the segmentation of pedestrians. RSS results are influenced by physical properties and motion states of different objects, which makes Depth-TRAP and SpatioTTRAP play their respective advantages. How further improve TARSS-Ne by combining object characteristies is also one of our future research directions.
E.3 Features visualization
Some feature visualization of core process in TARSS-Ne are shown in Fig. S6. It confirms that the proposed novel temporal information learning paradigm, i.e. TRAM, effective to make network learning pay attention to RSS target.
19

---

RD-View IoU (%)
RD-View Dice (%)
global
global
badagoornl
beckground
vchide
TARSS-Net D
vehicle
TARSS-NEUSS
sydlat
eydl
RAMP-CNN
padesslan
RSS-Net
podcstrian
20
190
20
-10
60
80
100
IoU (%6
Dice (%)
RA-View IoU (%)
RA-View Dice (%)
global
backgoondd
beckground
TARSS-NeL D
vchidc:
S
cyllst
TMVA-Net
RAMP-ONN
podostri
RSS.Net
20
100
80
100
(%)
Dice (%)
Figure S5: Comparison of class-wise performances.
(b)
(c)
(d)
Figure S6: Feature Visualization. (a) Input RD-view frame. (b) The activation response heatmaps of
TRAM outputs. (c) TARSS-Net outputs before Softmax. (d) Ground Truth Mask.
E.4
Visualization some examples
Some RSS results of TARSS-Net visualized in Fig. S7 More visualized performances provided in the supplementary videos.
Broader Discussions
Limitations. i) As can be seen in Table [ TARSS-Net performs worse than PKCIn-Net (which incorporates the advantages classical radar detection theories), T-RODNet and TransRSS (which enhance spatial feature representation based on Transformer) in RA view where the temporal infor- mation not obvious. ii) Deeply exploration on the learning principle and applicable scenarios of TARSS-Net_D D and TARSS-Ne is needed in future work. iii) TARSS-Ne makes good trade-off between real-time performance and RSS effects. However, there still room to improve its inference speed. These limitations also point to the improvement direction of TARSS-Net.
20

---

Pedestrian Vebicls Secuario
Pedestrian
mi Bicycle Scmmrrio
Tolesin
(4) Synchronized Camera Images
(b) Annotated Radar RD Representation
(6) RD Predictions by TARSS-Net
Figure S7: Visualization of some examples.
Social Impacts. As for radar detection, TARSS-Net is applicable to various practical applications, such as perceptions for autonomous driving, UAV surveillance and marine monitoring. Furthermore, the proposed novel paradigm of temporal relationship learning is causal and be exploited by more domains involving sequence data proceeding, including stock prediction, weather prediction, etc. However, inappropriate usage may lead decreased reliability, potentially resulting in safety and other issues.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper discusses the limitations of our work in Appendix F
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
For each theoretical result, this paper provides the full set of assumptions and complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper fully discloses all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The submitted supplementary materials provides the code with sufficient instructions faithfully reproduce the main experimental results. The dataset is publicly available and referenced in the core paper. The processing of raw data is described in Appendix A
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
peaips..c/public/u.ddees
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: This paper specifies all the training and test details
4.1 and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: This paper reports error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This paper provides sufficient information on the computer resources needed reproduce the experiments Appendix D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. cwwpurrisscccppppeeeee http://wwwpppp.cppp....//ccaaaaa
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss the potential societal impacts of our work in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators original owners of assets, used in the paper, properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "uV7GnVFqs8", "reviewer": "Reviewer_MgaH", "summary": "This paper proposed a temporal-aware framework, TARSS-Net, to enhance Radar semantic segmentation. The key idea is to propose a Temporal relation attentive module, TRAM (consists of Target-History Temporal Relation Encoding [TH-TRE] and Temporal Relation-Aware Pooling [TRAP] ), to capture the relations between Radar sequences.  Comparisons with baseline methods on three datasets (CARRADA, CARRADA-RAC and a self-collected KuRALS) show the advantages of the proposed method", "questions": "- How are the results when two frames are considered?\n- The performance decreases after 6 frames and increases again from 9 frames. Is there any analysis of that?\n- Can the authors provide to some feature visualization to confirm that whether the designed scheme payed attention to the object (or where to pay attention) in Radar sequences.", "limitations": "The authors appropriately discussed the limitations in the appendix.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and in details. The motivation of incorporating temporal information is practical.\n- The proposed method outperforms the SOTA methods in some aspects, especially in RD-View.\n- The experiments and ablation studies are extensive.", "weaknesses": "-   Since there are many existing works to incorporate spatio-temporal information, the technical contribution is limited without the insights considering the data characteristic of Range-Angle-Doppler data.\n- The performance increment is marginal, although various modules are designed.\n- It lacks of runtime statics to evaluate the time consumption introduced by temporal design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n\n**[W1. LIMITED TECNICAL CONTRIBUTION.]** Sorry to have caused the reviewer such concerns. The core point of this paper is to **redesign a better spatio-temporal modeling method for radar data from the perspective of temporal information utilization**. Indeed, there are many works in other fields that discuss the utilization of spatio-temporal information. However, as we analyzed in `L34~L44`, **there is still a large research gap in the related research of radar-oriented deep learning models**: \"In terms of temporal information utilization, the common practice is still 3DConv.\" Hence, we dedicated to exploring an advanced temporal modeling mechanism, whcih is urgently needed for modern radar (alleviating the problem of time-sensitive changes in radar data quality and learning better target representations). For the insights of RAD data handling, it is not the main focus of this paper, for two reasons: i) **TMVA-Net [21] had provided a multi-view learning approach that well balances information utilization and efficient processing for RAD data**; ii) **conducting detection/segmentation on RAD data is not universal in radar systems**, for some phased array radar system, it is more efficient to perform the detection directly on RD, e.g., KuRALS dataset used in this paper. In summary, this paper aims at the problem of temporal modeling which is more general in radar systems. From this perspective, this work summarizes the advantages and disadvantages of modern temporal modeling methods in `L82~L127` and the problems that need special attention when processing radar data in `L130~140`. We will reiterate our core research objectives in the Introduction section to highlight our contributions.\n\n**[W2. MARGINAL PERFORMANCE INCREMENT.]** Sorry not to surprise you, but **this is actually not a small improvement for RSS tasks**. As we mentioned in `L20~L24`, radar data is different from image data. It is susceptible to various interference and does not have semantic information, which make RSS more chanllenging. From the experimental part, it can be seen that the general semantic segmentation model of CV has not achieved good results on radar data (such as FCN, Unet). Then TMVA-Net [21], has improved the performance of RSS by about 2% (taking RD view as an example). The performance improvement of subsequent TransRadar [5], TransRSS [36] and PKCIn-Net [35] also increases by about 2%. To date, from FCN to TARSS-Net, RSS performance has improved from 66% to 75% on benchmark dataset, CARRADA. As you can see, **RSS performance is thus advanced bit by bit**. That said, **the authentic and reproducible 2% performance improvement** achieved by TARSS-Net is significant for the progress in RSS field.\n\n**[W3. LACK OF RUNTIME STATISTICS.]** Due to page limitation, the runtime performance comparison is listed in `Sec. E2 of Appendix`. TARSS-Net_D takes 43ms to infer one RAD@5Frames, 9 ms for one RD@5Frames and 9 ms for RA@5Frames (tested on single Nvidia3090).\n## Questions\n\n**[Q1. RESULTS WITH 2 FRAMES.]** Thanks for your suggestions to make our experiment more comprehensive and solid. We further tested TARSS-Net_D with 2 input frames on RD view CARRADA: mDice 73.7%, mIoU 62.0%, Precision 71.3%, Recall 76.5%. It is supplymented in `Figure 1 of attached PDF` for this rebuttal.\n\n**[Q2. ANALYSIS OF PERFORMANCE DECREASING AFTER 6 FRAMES AND INCREASING AGAIN FROM 9 FRAMES.]** This is a particularly valuable question. We did do some analysis of the trend of the curve shown in `Fig. 5`, and try to summarize some general conclusions that can guide TARSS-Net usage. Different operating conditions of radar affect the quality of each data frame, so in one RSS dataset, **it is hard to generalize how many historical frames are closely related to the target one**, or which historical frames are helpful to the current frame (poor quality history frame will introduce a lot of irrelevant noise and lead to performance degradation). Not to mention different RSS datasets. All networks dealing with temporal-sturctured radar data face the problem that they cannot calculate the correspondence between the number of input frames and network performance. **TARSS-Net has its own advantages to help ease the choice of input frame length**. With the design of TRIC layer, TARSS-Net could accept input with adjustable time length while keeping the number of parameters constant. When using it, readers can try and choose the number of input frames according to constraints of inference performance of the hardware (TFLOPS) without worrying about model size.\n\n**[Q3. Feature visualization.]** Please see `Figure 2 in attached PDF` for this rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FvyWzHPwnm", "reviewer": "Reviewer_awi8", "summary": "This paper primarily introduces a network model called TARSS-Net, designed for the task of radar semantic segmentation. It effectively utilizes the temporal information of radar signals by introducing a novel temporal modeling mechanism, enhancing radar semantic segmentation performance. Specifically, the paper proposes a module called TRAM for temporal relationship learning between the target and historical frames, integrating it with other core components to construct the TARSS-Net model.", "questions": "1.Can you provide a more detailed analysis of the computational complexity?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The author successfully applies time modeling to the radar semantic segmentation task, achieving state-of-the-art performance on the RD-View radar semantic segmentation benchmark.\n2.Rigorous ablation studies were conducted, providing solid evidence of the proposed method's efficacy.\n3.The paper provides a good review of time modeling, which is helpful to the radar semantic segmentation task.", "weaknesses": "1.This paper applies the time modeling to the field of radar semantic segmentation reasonably, but there are few innovations in the paper. \n2.The performance of time modeling approaches tends to degrade in crowded scenarios.\n3.It is difficult to significantly improve the performance of RA-View under the time modeling method\n4.The related works reviewed for the radar semantic segmentation task are not comprehensive.\n5.The placement of illustrations within the main text is not particularly reasonable, making re-display in the Appendix an unsatisfactory choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n**[W1. FEW INNOVATIONS.]** Thank you for recognizing the rationality of our approach. However, beyond reasonability, the novelty of TARSS-Net is also guaranteed. We feel sorry that we failed to make you see the novelty, as well as the careful thought and extensive experimental validation that went into it. Therefore, we think it is important to reiterate the real value and advantages of TARRS-Net. As we covered in `Sec. 2`, if you want to exploit temporal information in radar data, then you basically have 4 options: HMMs, RNNs, 3DConvs and Transformers. However, when we decide to use them, we encounter some unavoidable problems: i) **as shallow probabilistic models, HMMs are very limited in their ability to express primitives** for each single time step, and they **cannot be trained end-to-end** (`L86~L89`); ii) **RNNs** inherit the causal computing mechanism of HMMs and can be deeply trained end-to-end, but they **cannot fully enjoy the computational efficiency of parallelization** (`L90~L100`); iii) **3DConv** can be fully parallelized, but its calculation nature is non-causal and cannot be adaptively adjusted according to the length of the input sequence, i.e., **to process longer sequences, the network needs go wider or deeper**, which leads to the introduction of more training parameters (`L102~L114`); iv) the **Transformer** with SA as the core overcomes the problem of limited local receptive field when 3DConv deals with long sequences, and also overcomes the defect of RNNs that cannot be parallelized. However, its **computational complexity will show square level growth with the length of the sequence, resulting in computational redundancy** (`L115~L127`). Seeing these problems, we deeply realized that there is **NO FREE LUNCH**, we have to **redesign temporal modeling method for radar data, and RSS model should not be stagnant on these methods, it should have better sequence modeling methods**. To this end, we meticulously redesign the temporal modeling method and propose the design principles, resulting in **TRAM**, which **combines the advantages of causality, end-to-end learnability, constant model parameters under arbitrary length input, and linear growth of computational complexity with the length of the sequence**. Without careful design, we will not find a model in current methods that can achieve all of these advantages and still have the performance of SoTA. Therefore, we hope you to understand our efforts and recognize our work.\n\n**[W2. PERFORMANCE IN CROWDED SCENARIOS.]** The concern you raised is indeed worth discussing. In real-measured radar data, **target signatures often show extremely sparse characteristics** (this does not refer to the physical size of the target, but to the characteristics of the target reflected in the radar signal)，so **sparse and small target detection is a classic pain point which radar signal researcher** dedicated to solving. So, if we understand your concern correctly, the RSS task does not suffer from the problem of model performance tends to degrade in crowded scenarios, which is one of the reasons why TARSS-Net **emphasize the importance of learning and exploiting temporal relationships to enhance the representation of sparse target signature** for RSS.\n\n**[W3. DIFFICULTY OF PERFORMANCE IMPROVEMENT ON RA-VIEW.]** This is a good question that points out something unique to the RSS domain. The mismatch between the quality of RA-view data (for range and angle measurement) and RD-view data (for range and Doppler measurement) is a common phenomenon. Restricted by the hardware, most radars pay more attention to the ranging accuracy when designing, and the angular accuracy of the commonly used low-cost radar is hard to guarantee. Therefore, the difficulty in improving the segmentation accuracy of RA is more caused by the poor data quality, rather than the failure of the temporal modeling method. [35] notices this problem and modifies annotation quality of RA, resulting CARRADA-RAC dataset. Comparing `Table 1 and Table 2`, it can be seen that annotation correction can improve the performance of RA-view from 51.3% to 58.7 % (TARSS-Net_D). Therefore, it is reasonable to think that there is more room for the temporal modeling method to play a role after further improving the data quality.\n\n**[W4. NOT COMPREHENSIVE RELATED RSS WORKS.]** Temporal modeling is not an emerging topic in other fields. However, **effective and necessary modeling of temporal relationships has not been fully emphasized in RSS**. Therefore, in order to encourage readers to revisit the temporal relation modeling from the perspective of radar signal processing and understand the motivation of TARSS-Net, **we focus on the analysis of existing temporal modeling methods, the obstacles to their application in RSS domain, and the elaborate design of TARSSnet to face those obstacles**. It is believed that `Sec. 1 and 2` can bring somehow inspiration for readers in the NIPS community. It is worth noting that **in SoTA method comparison (see `Sec. 4.2`), we have included as comprehensive as possible the existing excellent RSS works** and made a brief analysis. Limited by the length of this paper, please visit the relevant index to download original paper for details of these works.\n\n**[W5. PLACEMENT OF ILLUSTRATIONS.]** Thanks for your kind reminder and we are sorry for the reading trouble caused by unreasonable placement. Due to the space limit of paper submission, we had to make some typography which might make it uncomfortable to read. **These will be corrected in next manuscript version, including the transposition of the order of Fig. 3 and Fig. 4, as far as possible to supplement the main text to reduce the redisplay in Appendix, etc**.\n\n## Questions\n**[Q1. MORE DETAILED ANALYSIS OF THE COMPUTATIONAL COMPLEXITY.]** There is real-time performance comparison in `Sec. E.2 of Appendix` and `Table 1 of PDF for rebuttal`, including model size, MACs, FPS and metrics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OpoUN5Ai9N", "reviewer": "Reviewer_WAmw", "summary": "In this work, the authors created a network called TARSS-Net for radar semantic segmentation. Compared to traditional methods, the authors emphasized the superiority of their approach in the clever utilization of historical information. Specifically, TARSS-Net incorporates a module called TRAM, which is designed similarly to the attention mechanism. This module first learns the relationship between historical frames and the current frame through the TH-TRE module, and then maps this learned relationship to hidden representations via TRAP. TRAM is specifically designed for radar semantic segmentation, ensuring that computational complexity remains manageable while achieving efficient and accurate radar semantic segmentation across three real radar segmentation datasets.", "questions": "Some questions are as follows:\n\n(1) What is the core innovation of this paper? Causal dilated convolutions also seem to align with the paradigm proposed by the authors. Compared to these simpler methods, what advantages does the method in this paper have? Why is it more advantageous?\n\n(2) Is it meaningful to discuss real-time performance for RSS semantic segmentation? If RSS semantic segmentation in most scenarios demands higher accuracy rather than efficiency, then the criticism of other methods in terms of efficiency in the paper seems less significant. Can the authors provide specific examples of scenarios to analyze and explain this?", "limitations": "The authors adequately discuss the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main advantages of the paper are as follows:\n\n(1) The paper provides a comprehensive review and analysis of the application background, existing methods, and challenges of RSS semantic segmentation.\n\n(2) The authors have significantly enhanced the model performance in RSS semantic segmentation tasks by balancing accuracy and efficiency through carefully designed methods, which holds practical significance.", "weaknesses": "629/5000\nThis paper may have the following potential shortcomings:\n\n(1) Lack of Clear Hypotheses and Reasoning: The authors designed a novel temporal modeling paradigm for RSS and claimed its effectiveness in the context of RSS semantic segmentation. However, they seem to provide their viewpoint directly without offering clear hypotheses and reasoning about why this paradigm is effective for RSS semantic segmentation, which may confuse the readers.\n\n(2) Insufficient Emphasis on the Design Motivation: The authors lack an emphasis on the motivation behind the model design. For instance, regarding the design of the TH-TRE module, the authors aim to use this module to encourage the model to focus more on high-dimensional information of the current time frame. However, Section 3.1 lacks an emphasis on this design motivation and focuses too much on formula statements. Given the considerable complexity of the designed method, it is crucial to introduce the design concept behind the module to the readers.\n\n(3) Issues with Paper Formatting and Visualization: There are some issues with the paper's formatting and visualization. For example, the sequence of Figures 4 and 3 does not align well with typical reading habits, leading to some reading difficulty. Additionally, some elements in the figures are not drawn rigorously; for instance, the input of the TH-TRE module should be a 4D tensor, but Figure 3 depicts it more like a 3D tensor. For readers accustomed to visual aids, the authors' lack of rigorous visualization might lead to a misunderstanding of the model details.\n\n(4) Potential Issues in the Experimental Section: There may be shortcomings in the experimental section, especially in comparison with the use of the Self Attention (SA) model in the temporal domain. Although the authors mention that applying SA in the temporal domain might lead to excessive computational consumption, the appendix shows that they have implemented a baseline model applying SA in the temporal domain based on ViT. Therefore, a more rigorous experiment should include this baseline model in the accuracy analysis (since accuracy analysis does not involve computational efficiency), as the authors have criticized this method in their analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n\n**[W1. LACK OF CLEAR HYPOTHESES AND REASONING.]** Sorry for the confusing, and you are right that clear hypotheses and reasoning are important. However, at present, the data-driven learning training of AI models allows researchers to conceptuate and design algorithms at a higher level (the functional and motivated level), which means the classification interface (Modeling functions of network) is no longer expressed in an explicit way by formulas, but is learned implicitly in the network parameters. It solves the problem that the classification interface cannot be shown by existing expressions. In this way, the algorithm performance has been further improved and generalized. **The writing of TARSS-Net exactly follows this theoretical system of priori interpretability.** Of course, **TARSS-Net essentially involves a series of theories and derivations of feature engineering, metric learning, differentiable layer design** and so on with radar signal as input. Due to the space limitation and ease of understanding for readers with general AI research background, this paper starts from the high level design and implementation, but the necessary derivations in key parts are preserved.\n\nIn addition, this paper gives a detailed introduction on why TARSS-Net  is effective for RSS. We have conducted a comprehensive discussion for existing time series modeling paradigms, and analyze their own drawbacks and the factors that make them unsuitable for RSS (see `Sec. 2`). Based on the above analysis, we illustrate the design motivation of TARSS-Net for RSS task one by one (see `L144-L155`), as well as detailed implementation methods in `Sec. 3`. Also, we further verify the superiority of TARSS-Net over the existing methods that consider temporal relation information.\n\nWe believe that **the confusion you mentioned can be eliminated after readers carefully read the full paper, the Appendix and the code**.\n\n**[W2. INSUFFICIENT EMPHASIS ON THE DESIGN MOTIVATION.]** In `Sec. 2`, we have conducted **a comprehensive discussion for existing temporal modeling paradigms**, and based on the above analysis, we illustrate **the design motivation of TARSS-Net for RSS one by one**, including the design motivation of TH-TRE module. We believe that after you go back and read the first two sections of this paper again, your questions will be answered.\n\n**[W3. ISSUES WITH PAPER FORMATTING AND VISUALIZATION.]** Sorry for the reading trouble caused by unreasonable formatting and visualization. Due to the space limit of paper submission, we had to make some typography which might make it uncomfortable to read. **These will be corrected in next manuscript version, including the order of Fig. 3 and Fig. 4, more rigorously draw for elements in the figures, etc**.\n\n**[W4. POTENTIAL ISSUES IN THE EXPERIMENTAL SECTION.]** Due to space limitation, we show the experimental results that can best help to verify the performance of TARSS-Net in the most concise way. **Due to the sparsity of radar targets, the dense computation of SA will inevitably introduce redundant computations on irrelevant information thus degrading the RSS performance**. The performance of the VIT model is supplemented in `Table 1 of attached PDF` for this rebuttal. We also promise to **add it to Table S2 in the Appendix** of the revised manuscript.\n\n## Questions\n\n**[Q1. THE CORE INNOVATION OF THIS PAPER.]** The core innovation is to propose an effective temporal modeling method specific to RSS tasks, i.e., the **plug-and-play TRAM which combines the advantages of causality, end-to-end learnability, constant model parameters under arbitrary length input, and linear growth of computational complexity with the length of the sequence**. These advantages cannot be satisfied in the same time when using other existing temporal modeling methods including Tranformers, 3DConv, RNNs and HMMs. For its significance, innovation and advantages, please read the first two Sections of the paper in details. In terms of **causal dilated convolution (CDC)**, it definitly meets the parallel-computation, larger RF with fixed-size kernels and causal computing mechanism, however **the dilation rate should be pre-defined**, i.e., if the input length changes, the hyper-parameter, dilation rate should be changed accordingly before training. While **TRAM does not require any adjustment when handeling different lengths of input**. Morover, as far as we know, **CDC is not in the 3D form which has the limitation for handeling temporal-spatial data** such as radar RAD sequence. Hence, instead of talking about CDC let's dive into 3DConvs, which are more prefered by the researchers in RSS field.\n\n**[Q2. IS IT MEANINGFUL TO DISCUSS REAL-TIME PERFORMANCE FOR RSS?]** Yes, it is very important to discuss the real-time capability of RSS. As a remote sensing device, radar is applied in many fileds, such as automatic driving, security warning and so on. Taking Ku-band drone surveillance radar as example, the PRT (pulse repetition time) is around 80us, and it has 128 coherent pulses in one CPI (one Range-Doppler frame), then the data rate for detection will be $\\frac{1 \\times 10^6}{80 \\times 128} = 97.66~\\text{FPS}$. This requires subsequent signal processing and detection/segmentation algorithms to match this data rate as much as possible. Hence, in order **to accurately detect and stably track the target, real-time performance is one of the important indicators of RSS task**, which has practical significance at the application level. Taking automatic driving as another example, the moving car needs real-time feedback of detection results in the surrounding environment, otherwise it will lead to unexpected consequences. Therefore, RSS needs to balance accuracy and efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rvzRbTGSnr", "reviewer": "Reviewer_kN5s", "summary": "This paper proposes a novel framework designed to integrate temporal information into radar-based semantic segmentation tasks. To achieve more effective temporal information integration, the framework introduces two key modules: the Target-History Temporal Relation Encoder (TH-TRE), which analyzes the relationships between different time frames, and the Temporal Relation Attentive Pooling (TRAP) module, which aggregates information along the temporal axis.", "questions": "1) Could the authors provide a comparison in terms of computational time?\n\n2) Although the primary baseline, PKCIn-Net, also incorporates temporal information, the difference in performance between the proposed algorithm and PKCIn-Net appears to be minimal.\n\n3) Additionally, the authors mention that this paper considers compression from both spatial-temporal and depth-temporal dimensions. Therefore, a pertinent question arises: would it be possible to first fuse spatial and depth information, and then subsequently fuse this combined data along the temporal axis?", "limitations": "Please consult the question section for further information.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a novel algorithm for radar-based semantic segmentation tasks. The manuscript is well-constructed, with the discussion section clearly articulating and comparing the differences between various algorithms. The newly designed algorithm demonstrates a notable improvement in performance without increasing the model size.", "weaknesses": "Please consult the question section for further information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# **Questions**\n\n**[Q1. COMPUTATIONAL TIME COMPARISON.]** There is real-time performance comparison **in `Sec. E.2 of Appendix`**, which shows the comparison between TARSS-Net and some other models, involving**model size**, **calculation amount**, **real-time performance** and other information of in details. In addition, we also add the VIT model in`Table S2` as a baseline for comparison, which can also be found in`Table 1 of attached PDF` for this rebuttal.\n\n**[Q2. MINIMAL DIFFERENCE IN PERFORMANCE COMPARED WITH PKCIN-NET.]** PKCIn-net makes the classical radar target detection CFAR principle deeply learnable, and proposes a radar-specific target detection operator PeakConv (PKC). **The focus of PKC is not on incorporating temporal information, but we include it in the comparison as it is a landmark work on introducing deep learning to RSS.** Under the premise of the same model size, TARSS-Net outperforms PKCIn-Net by an average of 2% (on all data-view and three dataset). First of all, given the particularity and difficulty of RSS tasks, **a 2% performance improvement is significant**, we also illustrate in our responses to R. MgaH how the RSS performance has improved little by little (if you want to learn more, please see response of `W2 to R. MgaH`). Secondly, PKCIn-Net focuses on making the classical radar detection theory learnable, and TARSS-Net focuses on introducing an efficient temporal relationship modeling method suitable for radar signals. In the later work, **we will also consider adapting PKC to TARSS-Net to further improve the performance of the model**, since TRAM and PKC are not in conflict due to their different focuses. Both PKCIN-Net and TARSS-Net point the way for the development of the RSS field. That is, compared with directly applying the mature learning method in other field (such as CV, NLP, etc.), designing the learning paradigm suitable for the characteristics of radar signals is more beneficial.\n\n**[Q3. WOULD IT BE POSSIBLE TO FIRST FUSE SPATIAL AND DEPTH INFORMATION, AND THEN SUBSEQUENTLY FUSE THIS COMBINED DATA ALONG THE TEMPORAL AXIS?]** This is a particularly valuable question. When we originally designed the TRAP module to implement temporal relation measurement and weighted-fusion, we intended to incorporate spatio-temporal information at the same time. However, for the space and depth with high dimensions, it is the most economical choice to do dimensionality reduction and fusion processing separately, i.e., Spatio/Depth-TRAP. **If we want to fuse the two dimensions at the same time, it is bound to introduce modules with larger parameters, so we did not explore it in this work.** Therefore, in the subsequent network optimization process, **considering the complexity** of RSS network to facilitate its application, **we form two versions of Spatio-TRAP and Depth-TRAP on the premise of ensuring their performance advantages**. Your idea is very meaningful, and as mentioned in the `limitations Section in Appendix F (L1017)`, deeply exploration on the learning principle and applicable scenarios of TARSS-Net_D and TARSS-Net_S is needed in future work. Based on this, we look forward to working out an exciting solution to your pertinent question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our respect to all reviewers and AC. Thanks for your time and hard work. Based on your professional opinions, we have carefully replied all the high-value questions and supplemented the content accordingly.

Meta-review: This paper receives an overall rating of 4/5/5/5 (average: 4.75). After the rebuttal, two positive reviewers were generally satisfied with the rebuttal and chose to maintain their ratings, while two other reviewers did not respond. The AC read the paper and all discussions. Although the technical contribution is not significant, it was well-prepared and improved the accuracy of radar-data segmentation. Additionally, the AC thinks the main issues raised by the reviewers have been addressed. So, the AC recommends accepting this paper, and urges the authors to add the supplementary contents during the rebuttal to the final paper.
Decision: Accept (poster)