TARSS-Net: Temporal-Aware Radar Semantic Segmentation Network

Authors: youcheng  zhang
liwen  zhang
zijun  hu
pengcheng  pi
teng  li
yuanpei  chen
shi  peng
zhe  ma
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:33:09

Abstract: Radar signal interpretation plays a crucial role in remote detection and ranging. With the gradual display of the advantages of neural network technology in signal processing, learning-based radar signal interpretation is becoming a research hot-spot and made great progress. And since radar semantic segmentation (RSS) can provide more fine-grained target information, it has become a more concerned direction in this field. However, the temporal information, which is an important clue for analyzing radar data, has not been exploited sufficiently in present RSS frameworks. In this work, we propose a novel temporal information learning paradigm, i.e., data-driven temporal information aggregation with learned target-history relations. Following this idea, a flexible learning module, called Temporal Relation-Aware Module (TRAM) is carefully designed. TRAM contains two main blocks: i) an encoder for capturing the target-history temporal relations (TH-TRE) and ii) a learnable temporal relation attentive pooling (TRAP) for aggregating temporal information. Based on TRAM, an end-to-end Temporal-Aware RSS Network (TARSS-Net) is presented, which has outstanding performance on publicly available and our collected real-measured datasets. Code and supplementary materials are available at https://github.com/zlw9161/TARSS-Net.

Main paper: 
 introduction: 
 Radar is a reliable remote sensing device for its robustness in adverse weather and illumination conditions. It is widely used for applications like the autonomous driving [33, 21, 36] , Unmanned Aerial Vehicle (UAV) surveillance [9, 19] , sea monitoring [31, 24, 32] , etc. However, the received scattered radar signals are high-entropy information body coupled with environmental clutters from a large spatial range, device noise and moving target information (i.e., distance/range, direction/angle and velocity/Doppler shift). These components are additionally condensed into the received echoes, which is inherently a gap in understanding of radar signals to human perception, as opposed to visible light images, which are modalities aligned with human visual perception, and natural language texts that are naturally based on human semantic understanding. This human-unfriendly perception gap causes great difficulty in semantic perception of scenes and objects [22] . More information on radar signals and processing is in Appendix A.Encouraged by the success of deep learning techniques in computer vision, in especial, object detection [11, 17, 27, 26] and semantic segmentation [20, 28, 2, 10] , some efforts have been made recently to better understand complex radar data. Compared with detection models [33, 6] using bounding boxes, the segmentation models [1, 14, 21, 36, 35] can provide pixel-wise (a unit of the range-angle (RA) or range-doppler (RD) frequency representation) detection results for objects and even the background, which is practical for radar scene understanding and is also followed by this work. Most of these methods are based on convolutional auto-encoding-decoding (CAED), which take the frequency representations of a series of fast Fourier transforms (FFTs) on the radar signals as input, and make predictions on the RA or RD view or both views.By analysing current efforts in RSS, two primarily concerned problems can be concluded for effective design of RSS system. i) How to capture target signature effectively in spatial domain. Typically, atrous spatial pyramid pooling (ASPP) [3] is used to obtain multi-scale spatial information [14, 21] , and deformable convolution is utilized to extract target features with irregular shape [6] . Specifically, PeakConv (PKC) [35] is the learning-based operator tailored for radar signals which aims to capture target signature's salience. ii) How to utilize inherent temporal information of radar input tensor. Commonly, 3D convolution (3DConv) is implemented [21, 33, 6] . It is worth noting that, compared with the direct use of existing methods, taking into account the intrinsic merits of radar signal can achieve more performance advantages. In terms of temporal information utilization, the common practice is still 3DConv. Few work has explored a specific temporal modeling mechanism for RSS and even less conducted a discussion of other off-the-shelf methods from the perspective of RSS task.Considering the above research gaps, this paper conducts a comprehensive discussion from the perspective of RSS for existing time series modeling paradigms, including i) causal temporal relation modeling, represented by hidden Markov models (HMM) [25] and recurrent neural network (RNN) family [8, 7, 4] ; ii) parallelized sequence representation modeling, represented by 3DConv [29] and Transformer [30] family. Based on this, we propose a novel modeling paradigm for radar temporal information utilization. Specifically, this paper makes the following efforts: i. A novel temporal modeling paradigm for RSS. Based on the in-depth analysis and discussion of existing temporal modeling methods, several specific design principles for RSS temporal modeling paradigm are given ( § 2). Following these principles, a novel temporal information learning paradigm for radar spatio-temporal tensors is proposed, i.e., data-driven temporal information aggregation with learned target-history relations.ii. Temporal relation attentive module (TRAM). TRAM is proposed to realize the learning paradigm mentioned above, which is a flexible temporal relation encoding and aggregating module. For encoding, the target-history temporal relation encoder (TH-TRE) is designed, which aims to capture relations between target frame and its historical neighbors. For aggregating, temporal relation attentive pooling (TRAP) with two forms, i.e., learning in temporal-depth and spatio-temporal, is presented. Both the encoder and aggregator can be independently inserted into arbitrary 2/3DConvbased networks ( § 3).iii. Reasonable-scale RSS Network with variable input time lengths. Processing high-dimensional radar sequence while ensuring the use of temporal information is computationally expensive. To ensure RSS model has the ability of online real-time processing, temporal-aware RSS network (TARSS-Net) with moderate parameters is designed based on TRAM, which keeps its parameter scale constant while accepting adjustable time length input. Compared with existing temporal modeling methods, TARSS-Net could effectively avoids the long-term dependence decreasing risk as well as gradient disappearance problem, which shows good segmentation performance and inference efficiency in both multi-view and single-view conditions ( § 3, Appendix C). iv. Model performance verification with real-measured data in different detection scenarios. To verify the scope of application of TARSS-Net, we conduct quantitative experiments on different real-measured large scale radar datasets including CARRADA [22] which is collected from a low cost FMCW (≈ 77GHz) on-board radar in driving scenario and self-collected dataset, KuRALS, recorded from a Kurz-under (Ku) band (≈ 17GHz) radar for UAV surveillance and sea monitoring. Experimental results show TARSS-Net can achieve state-of-the-art (SoTA) performance ( § 4). • HMM [25] is the classic causal temporal modeling methods, which uses hidden states to model temporal dependencies as shown in Fig. 1-(a1 ). Based on the assumption of Markov first-order homogeneity and observation independence, it can conveniently use the transition probabilities between hidden states to describe the intrinsic causal relationship of sequential data. However, as a shallow probabilistic model, HMM cannot perform data representation and downstream task prediction end-to-end. Meanwhile, the Markov assumption and discretized encoding of the hidden states also limit its ability to describe long-term dependencies. • RNN [8] also introduces hidden state to describe temporal relation of input sequence similar with HMMs. However, its hidden state is extended to a continuous vector, i.e., real-valued representation projected by a parameterized connection shown in Fig. 1 -(a2). In this way, RNN is no longer limited to a probabilistic model, but a learnable one that can be deepened. The gradient descent algorithm can be used to realize end-to-end training of RNN's representation and prediction, which greatly alleviates the problem of HMM's insufficient ability to depict long-term dependencies. With more complex computing units such as LSTM [12, 7] and GRU [4] , its representational power can be further enhanced. However, in addition to the resistance caused by gradient dispersion/disappearance under the back-propagation framework, RNNs are sequential causal computing model after all, and their popularity is gradually fading under current main theme of multi-core parallel computing framework.ii. Parallelized sequence representation modeling.• 3DConv [29] uses the concept of local receptive field (LRF) and the design of shared kernel. It can efficiently fulfill encoding and representation of high-dimensional spatio-temporal tensors in a multi-core computing environment as shown in Fig. 1 -(b1). Therefore, 3DConv has become the preference for RSS models that require spatio-temporal encoding capabilities [14, 21, 33, 6] . However, due to the limitation of LRF, it cannot achieve temporal-dependence across long-term range while retaining efficient computation and appropriate parameter amount. Meanwhile, an inherent contradiction between 3DConv and RSS is ignored by most of current works, that is, convolution naturally follow the rules of context encoding, which is more suitable for obtaining responses in the center position of LRF. For input sequence,{x 1 , • • • , x 1+τ , • • • , x 1+2τ } ∈ R H×W ×(2τ +1) , performing predictions on x 1+τ is more of a natural advantage of 3DConv-based models, instead of predicting x 1+2τ as most RSS required. In terms of pragmatism, performing the prediction at the current instant, i.e., (1 + 2τ )-th time step, is more reasonable, more causal, and more real-time. For this reason, 3DConv might not be optimal for RSS. • Transformer [30] is becoming the backbone of fundamental models in many computing fields [23, 16] . Compared with RNN, it overcomes the problem of parallel computing for handling sequential data; compared with convolution, it breaks the limitation of LRF by preserving the sequence-tosequence encoding style. The key to Transformer's success lies in self-attention (SA) mechanism. However, its problem is also with SA. By using parameterized connections with different weights, the sequence {x t } T t=1 is abstracted into the forms of {q t } T t=1 , {k t } T t=1 and {v t } T t=1 , and then the temporal relation between any two primitives is obtained by densely calculating the pairwise inner product along time dimension, i.e., {q i k ⊤ j } T,T i=1,j=1 , resulting the computational complexity of O(T 2 ) for temporal relation modeling. This greedy manner is acceptable for data where only time-dimension is a concern, but for spatio-temporal tensors with large spatial ranges, e.g., radar data, the computational cost versus performance gain ratio may not be cost-effective. Moreover, since predictions are required only on current time step, computing resources should not be blindly allocated to each historical time step of input, which would lead to redundancy.The above analysis motivates this paper to redesign spatio-temporal encoding module for RSS task, and teases out the following 5 design principles:• Utilizing parameterized rather than probabilistic connections to characterize the temporal relations of sequence like Transformers and RNNs do.• On the premise of making predictions at current time step, the module should emphasize the use of the current input frame, i.e., the non-context calculation in time.• The module should be able to handle temporal relations in parallel.• Considering the high-dimensional spatio-temporal characteristics of radar data, the module should ensure the efficient learning ability of long-term relationship and keep the parameters appropriately scaled.• Due to the coupling of device noise and environmental clutter, radar data would be non-smooth in time dimension. In this way, the module should consider the contribution of each time step differently during historical information aggregation.To take into account the above principles, a novel temporal learning paradigm for RSS is proposed as shown in Fig. 1-(c ), i.e., data-driven temporal information aggregation with learned target-history relations. The careful designs are as follows:• The idea of modeling temporal relations with parameterized connections in RNNs and Transformers is preserved, i.e., temporal relation is explicitly treated as an intermediate embedding obtained by a parameterized layer/block (relation embeddings mentioned in § § 3.1).• The current/target-history relation encoding mechanism is introduced (see TH-TRE in § § 3.1). In this way, the complexity of O(T 2 ) for temporal relations in SA is avoided (O(T ) for TRAM), the use of the current frame is emphasized, and the convolution is non-context calculation in time.• In order to achieve temporal relation encoding in the parallel way, temporal-relation-inception convolution is designed in TH-TRE (TRIC mentioned in § § 3.1), which is also helpful to keep model scale constant with variable time length of input sequence.• A learnable temporal pooling layer (see TRAP in § § 3.2) is designed to measure and reallocate the contribution degree of different target-history relation embeddings, thus the adverse effects caused by time non-smoothing of radar data quality can be alleviated in a data-driven way. 
temporal relation attentive model (tram): 
 As illustrated in Fig. 2 , the proposed TARSS-Net is based on CAED framework, which consists of basic encoder, TRAM, latent space encoder (LSE) and decoder. For each single-view radar input sequence, the basic encoder is used to generate high-level representations. The LSE is used to align and fuse the high-level semantic features of different views, which is further applied to the single-view decoder to improve its performance. Decoder receives inputs from TRAM and LSE, and finally produces segmentation results on RD and RA perspectives, respectively. As the key of temporal relation learning, TRAM will be detailed in this section, and remaining components can be found in Appendix C. TRAM contains of two components: the target/current-historical temporal relation encoder (TH-TRE) and temporal relation-aware pooling (TRAP).TH-TRE aims at capturing temporal relations of encoded target frame and its adjacent historical frame features. To achieve this goal, we design the temporal-relation-inception convolution (TRIC) block to handle each target-historical feature pair, as shown in Fig. 3 . Given the feature map sequence {X t-τ , . . . , X t-1 , X t } ∈ R C×(τ +1)×H×W obtained from a basic encoder (e.g., RA Encoder), the whole process of TH-TRE can be formalized in Eq. (1). C, (τ + 1), H, W are the numbers of channels/depth, time frames, range cells and angle cells, respectively.EQUATIONWhere K 1 (•) : R C×H×W → R C×H1×W1 and K 2 (•) : R 2C×H1×W1 → R C×H2×W2 are 2D convolution layers, the operator ⊕ D and ⊕ T denotes concatenation on depth and temporal dimension, respectively, Max is the 2D max-pooling operation with the spatial downsampling rate of 2. In this work,H 1 = H/2, W 1 = W/2, H 2 = H/4, W 2 = W/4.It can be seen that K 1 (•) drags input feature maps of different time frames into a common feature space to ensure representation compatibility [34] , and then K 2 (•) takes the enhanced feature maps of each compatible feature pair, i.e., K 1 (X t ) ⊕ D K 1 (X i ), as input to learn the relations between target frame and its historical frames. In this way, the associations between target feature maps, X t , and each of its historical feature maps, {X i } t-1 i=t-τ is obtained. Finally the outputs from TRIC will be grouped together with K 1 (X t ) to form the output relation embeddings, i.e., { Xt-τ , . . . , Xt } shown in Fig. 3 , It is worth noting that convolution kernels in TH-TRE are shared, which helps it accept input with adjustable time length while keeping the number of parameters constant. The relation embeddings obtained by TH-TRE block are still in a temporal sequence form. While, for making final predictions on the target frame, the aggregation for generating summarized representation of entire input sequence is required. To this end, a learnable temporal pooling, i.e., TRAP, is proposed. In general, the TRAP block aims at perceiving the contribution degree of each historical frame for prediction task according to the target-history relations, and using these measurements of importance to aggregate the temporal information in each single-view radar sequence. However, due to the computational pressure and parameter expansion caused by high-dimensional spatial-time tensors, it is necessary to consider the trade-off of information utilization in each dimension before aggregation. In turn, two forms of TRAP are presented, i.e., Spatio-TRAP and Depth-TRAP. For each form of the TRAP, there are two main steps: compression (discard) and attentive pooling (aggregation).Spatio-TRAP. Spatio-TRAP is performed on the entire spatial domain of input feature maps. Therefore, the importance of temporal relations will be estimated on the spatial space of relation embeddings.As shown in Fig. 4-(a) , given the relation embeddings, Xit i=t-τ ∈ R C×(τ +1)×H2×W2, the two main steps of Spatio-TRAP are as follows:i. Depth-Compression. First, all the relation embeddings are compressed by both global depth average-pooling and max-pooling. Then a temporal-shared 2D convolution block is used to generate the contribution map sequence, {M i } t i=t-τ ∈ R 1×(τ +1)×H2×W2 , for following aggregation step. For each M i , the process can be formalized as follows,EQUATIONWhere Avg/Max DC (•) denotes the global depth average/max-pooling operation, K ST 1 (•) : R 2×H2×W2 → R 16×H2×W2 and K ST 2 (•) : R 16×H2×W2 → R 1×H2×W2 are temporal-shared convolution layers with kernel size of 3 × 3 and 1 × 1, respectively.ii. Spatio-Temporal Attentive Pooling. To modulate the input feature maps in spatio-temporal dimension, a 2D softmax is performed on each map of {M i } t i=t-τ . Moreover, as deriving from the relation embeddings ( Xit i=t-τ), the contribution maps have a smaller spatial size than{X i } t i=t-τ , i.e., [H 2 , W 2 ] = 1/4[H, W ].Hence a spatial interpolation operation for expanding the size of {M i } t i=t-τ is required, so as to ensure the dimension consistency of the following weighted summation and skip connection. Then the whole process of this step can be formalized asXST t = t i=t-τ W ST i ⊙ X i,d C d=1 + X t ,where,EQUATIONWhere, XST t and W ST i are enhanced target representation and spatio-temporal contribution weights of the i-th frame. Since the nearest interpolation Intp(•, [•, •]) expands HW/H 2 W 2 times the spatial size of 2DSoftmax(M), the values of W ST i should be shrank to satisfy τ +1,H,W i=1,h=1,w=1 w ST i,h,w = 1. Depth-TRAP. Depth-TRAP is performed on the depth of input feature maps. During learning process, features expanded by the depth dimension are of great significance for unit-level classification (semantic segmentation), and they adequately represent semantic information. In this way, Depth-TRAP measures the importance of temporal relations on semantic space of relation embeddings. Similar with the spatio form, Depth-TRAP also requires compression and aggregation steps (see Fig. 4-(b )), which are as follows:i. Spatio-Compression: The input relation embeddings are first compressed in spatial domain by global average-pooling and max-pooling, then these two global pooled sequences are concatenated and presented to a temporal-shared small MLP network, G SC (•) : R 2C → R C , to obtain the spatiocompressed sequence, x SC i t i=t-τ . This process can be formalized as follows,EQUATIONWhere Avg/Max SC (•) denotes global average/max-pooling operation on spatial domain, with a kernel size ofH 2 × W 2 .ii. Depth-Temporal Attentive Pooling: This step calculates the contribution vector for each spatialcompressed temporal relation embedding, x SC i , through depth dimension, which achieves the importance measurement of temporal relations in semantic space. Then, with these contribution vectors, a weighted summation along with a skip connection is conducted to aggregate each single-view encoding feature sequence, {X i } t i=t-τ , and therefore the enhanced target representation, XDT t ∈ R C×H×W is obtained. The contribution vectors can be obtained as follows,W DT = Softmax G DT (v i ) t i=t-τwhere,EQUATIONWhere G DT (•) : R C → R C denotes the temporal-shared MLP, and p i,d is the depth-temporal position encoding result following self-attention [30] for each element of x SC i t i=t-τ . Similar to Spatio-TRAP, the contribution vectors are scaled by softmax to jointly modulate the input feature maps in both depth and time dimensions. Then the scaled vectors, W DT ∈ R C×(τ +1) , are used to aggregate information from input feature maps as follows,EQUATIONWhere the operator ⊙ denotes Hadamard product, and x i,h,w ∈ R C is the feature in each spatiotemporal position of input feature maps {X i } t i=t-τ obtained from a basic encoder. Finally, the output target representation, Xt , enhanced by the TRAM will be presented to the corresponding decoder. 
datasets and training setup: 
 Three datasets have been used to valid the performance of TARSS-Net including: CARRADA, which contains multi-view annotated radar recordings (RAD tensors) for 4 categories of objects in driving scenarios under different weathers conditions; CARRADA-RAC [35] , which is an improved version of CARRADA with calibrations on RA view; KuRALs, which is a real-measured dataset with refined annotation collected by a Ku-band surveillance radar and includes various typical targets.The training setup for TARSS-Nets and other compared SoTA networks strictly follows the consistent configuration, including hardware platform, hyper-parameter settings and evaluation metrics, and the number of input frames for TARSS-Nets is 5 by default (τ = 4). See Appendix D for more details.The most widely used RSS dataset, CARRADA, is used to comprehensively compare the performance of TARSS-Net with existing RSS methods. CARRADA-RAC and KuRALS dataset are utilized to explore the generalization and stability of TARSS-Net, on which only representative methods are compared, i.e., TMVA-Net (temporal-based model) and PKCIn-Net (current SoTA). TARSS-Net_S and TARSS-Net_D denote the TARSS-Net with Spatio-TRAP and Depth-TRAP, respectively. i. CARRADA. The overall results on the test subset are listed in Table 1 . Generally, the models that explicitly consider the temporal dynamics in radar signals, e.g., TMVA-Net [21] , can obviously achieve better performance than other convolution-based models [20, 28, 2, 14, 6] . By introducing SA mechanism in Transformers [30, 18] into convolution frameworks, TransRadar [5] , T-RODNet [13] (the RD scores are not provided in the literature) and TransRSS [36] (the code is not public in the literature) further enhance the representation ability in spatial dimension, thus pushing the performance of RSS to a new height. However, due to the explosion of parameter scale caused by the high-dimensional radar tensors, none of them use SA in temporal domain. Compared with the existing RSS methods using temporal information, TARSS-Nets with the well-designed temporal learning paradigm exactly follows the 5 design principles mentioned in § 2 and achieves the best overall performance (average metrics for both views) with reasonable parameter scales. Unfortunately, its RA performance is not so prominent, as the RA data does not change much over time or poor angular resolution, resulting in less obvious temporal information that can be exploited. Specially, PKCIn-Net [35] does not belong to the category of temporal information modeling, but its better performance in the RA view shows that it is helpful to introduce the classical radar signal processing theory into RSS models, which provides a direction for the future improvement of TARSS-Nets.ii. CARRADA-RAC. Models listed in Table 1 fail to aquire the ideal performance on the RA view. One main reason is the poor angular resolution of common low-cost FMCW radars. CARRADA-RAC [35] has calibrated the RA annotations, which could give a more realistic performance comparison for different models. As shown in Table 2 , the global performance are improved after annotation refinement, especially in RA view and TARSS-net still achieves the performance of SoTA.iii. KuRALS. This dataset is different from CARRADA in terms of radar systems and detection scenarios, which could complement testing the generalization and stability of models. Experiments in Table 3 show that TARSS-Net cloud maintain its superior performance in different application scenarios. The model marked with sv has been modified to a single-view version as needed. Three sets of ablation experiments are conducted to verify the effectiveness of proposed TRAM module and its two independent blocks, TH-TRE and TRAP. Depth-TRAP is used as default. Ablation Experiments on TRAM. As control experiments, two baseline models are designed by incorporation different mechanisms of temporal information utilization to TARSS-Net: i) Baseline-A: replacing TRAM from TARSS-Net with a single temporal global average/max-pooling (GAP/GMP) layer; ii) Baseline-B: replacing TRAM from TARSS-Net with temporal aggregation layers designed in TMVANet [21] , which consists of stacked 3DConvs. Results show in Table 4 .Baseline-B achieves better performance on both views than Baseline-A, which indicates that parameterized learnable aggregation has obvious advantages over parameter-free (max or avg) aggregation. While by using the proposed TRAM, TARSS-Net can obviously exceed 3DConv-based Baseline-B on almost every evaluation metric, which fully demonstrates the effectiveness of the proposed temporal-aware learning paradigm for RSS networks.The Effectiveness of TH-TRE. TH-TRE aims to capture temporal relations of target frame and its historical frames. To verify its effectiveness, two more contrast models are presented: i) Baseline-A w/ GAP & TH-TRE: adding TH-TRE block to Baseline-A w/ GAP, in which a single temporal GAP layer is used to aggregate the temporal relation embeddings obtained from TH-TRE; ii) TARSS-Net w/o TH-TRE: TARSS-Net without explicitly learning temporal relations of the target frame and its historical frames. Here, the TRIC block is replaced with a regular 3DConv block, which contains two convolution layers with the same kernel size of 1 × 3 × 3 and stride of (1, 2, 2). As shown in Table 5 , the addition of TH-TRE block is beneficial to improve RSS performance of both Baseline-A w/ GAP and TARSS-Net w/o TH-TRE. This block realizes current/target-history relation encoding under the premise of maintaining temporal order. In this way, TH-TRE emphasizes the use of current frame and achieve more computationally efficient and powerful temporal encoding capacity than simple temporal compression mechanism such as GAP and 3D convolution. The results listed in Table 6 show that either form of TRAP block can aggressively improve the performance of both Baseline-A model and TARSS-Net. This illustrates that different history frame has unequal closeness with target frame, and TRAP would be an efficiently way for reassigning weights to temporal relations aggregation, which effectively alleviates the negative influence on RSS caused by temporal unsmoothness of low-quality radar data. In addition, Depth-TRAP-based models always obtain better results than Spatio-TRAP-based models. Such performance gap may caused by the aggressive compression of the semantic information (reflected in the channel dimension of features) in Spatio-TRAP block. As mentioned in § § 3.2, Depth-TRAP pays more attention to the semantic features obtained from the TH-TRE block, and such consideration makes the network generate more informative representations than Spatio-TRAP. Therefore, we argue that maintaining more semantic information might be an effective way to further improve Spatio-TRAP, which is worth of investigating in the future work.The RD performance of TARSS-Net_D increases with time length of inputs grows and achieves the optimum at 6 frames, i.e., τ = 5, as presented in Fig. 5 . With the careful design of temporal modeling paradigm, TARSS-Net can handle arbitrarily long sequence relations without increasing parameter scale, which is significant to find the optimal duration of temporal dependencies not only for RSS tasks but also for all temporal relation modeling methods.In addition to model scale, the computational complexity and inference speed of the RSS methods are also important for deploying applications, which is measured by multiply-accumulate operations (MACs) and frames per second (FPS), respectively. All the real-time performance shown in this section are obtained on a single RTX 3090 GPU.Among existing RSS models, TMVA-Net is the SoTA one for multi-view temporal relation learning. As far as our knowledge, none of existing methods applied Transformer or SA on the time dimension in RSS task because of the exploding number of parameters. In order to deeply explore the real-time performance of different temporal relationship modeling methods, we introduce vision Transformer (ViT) into the basic architecture of TARSS-Net, which forms a ViT-based baseline method, called Vit-based-Net. The left part of Table 7 compares the real-time performance of TARSS-Net with TMVA-Net and Vit-based-Net with multi-view inputs. All methods take 5 consecutive time frames as input. It can be seen that directly introducing Transformer in temporal relationship modeling of radar data is inefficient, which would consume more computation source but obtaining lower RSS performance. TARSS-Net consumes more computation and inference time than TMVA-Net in temporal relation learning, but achieves better RSS accuracy. In addition, the inference speed of 23 FPS in TARSS-Net_D is also sufficient for real-time applications. The right part of Table 7 compares real-time performance of several temporal relationship learning models in single view. That is, the results for RD and RA views are measured separately by the corresponding single-view network versions. In terms of real-time performance, TMVA-Net still has the fastest inference speed. But Vit-based-Net achieves the worst inference speed with the most number of parameters. However, it is not to be overlooked that TRASS-Net still leads the pack in terms of RSS performance with competitive real-time performance.More supplementary experiments are presented in Appendix E, including ablatioin experiment of AD encoding branch, class-wise performance comparisons, features visualization of core process in TARSS-Net and some example visualization of segmentation results. to say, temporal modeling only plays a limited role in these works. However, in this work, we dedicate to this temporal information utilizing problem, and attempt to define a general learning paradigm: capturing temporal relations first and then aggregating them by a task-driven manner. These relations are extracted from target frame and its historical neighbors in an explicit way, then a learnable relation aggregator would estimate the importance of those relations and group them adaptively into a single representation according to the task. It is different from previous works which merely rely on convolutions and learning-free aggregating operators, e.g., max/avg-pooling.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: This paper fully discloses all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper. 
conclusions: 
 As mentioned in § 1, the existing temporal modeling methods can be summarized into two categories: i) Causal temporal relation modeling, represented by HMM and RNN family; ii) Parallelized sequence representation modeling, represented by 3DConv and Transformer (SA) family. Despite the advantages of these methods, directly using RNN and SA would introduce other problems. For RNNs, the dependencies between elements are captured by a recursive dependency calculation manner. That is to say, to produce an output, O t at t-th frame, the former one O t-1 should be generated first. Such recursive dependency encoding mechanism could not fully enjoy the convenience of parallel computing. For SA, the dependencies between elements are captured across each pair of elements in the input sequence. However, extracting all context relations could cause redundancy. Since our task is to make prediction on target frame, it is reasonable to pay more attention to target-history pairs instead of each pair. Moreover, SA is more friendly to MLP-based frameworks, but for convolutional networks, its multi-scale spatial encoding capacity would be restricted to some extent since there is a trade-off between information compression and parameter scale when flattening the spatio-structured feature maps into a vector. Comparatively speaking, the proposed TRAM module well overcomes the above shortcomings. It can capture the temporal relationship of variant input time length without increasing scale of network, and at the same time retain the advantage of parallel computing.To intuitively show our basic idea and parallel computation of proposed TRAM, we drew Fig. S1 . One implication can be seen in the figure, that the importance measurement can be regarded as a soft-selection for target-history relations. It actually drives the model to focus on the useful part of history information. That is to say, if one can decide which historical frames contribute most to target frame representation and prediction, it would be easier to design a model with a longer temporal dependence and less learnable parameters, since history information with less importance could be ignored directly. And we argue that filtering useful historical information in advance is also worth to investigate in the future, so that the model can "see more history", while keeping a reasonable parameter scale. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The paper discuss the potential societal impacts of our work in Section F. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets, used in the paper, are properly credited and the license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: New assets introduced in the paper are well documented and the documentation is provided alongside the assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: As illustrated in Fig. S2 , the proposed TARSS-Net is based on CAED framework. In addition to TRAM, some core components are introduced as follows.Basic encoder. For each single-view radar input sequence, the basic encoder is used to generate highlevel representations, and multi-view radar sequence can be handled by performing these encoders in parallel. All three single-view encoders are designed with the same structure for simplicity. Each encoder is mainly composed of two 3D convolution blocks with kernel size of 1 × 3 × 3, which shares convolution kernel learning of each frame on the time dimension, so as to realize feature interaction between all historical and target frames. It is worth noting that this is different from using 2D convolutions, which do not enable parameter sharing and feature interaction in dimensions outside the 2D plane. To further compress the spatial size of feature maps, each convolution block follows a spatial max-pooling layer.Latent Space Encoder (LSE). The basic encoder and TRAM are performed on each single-view of the radar sequence. However, one problem in multi-perspective learning structure is how to extract the rich knowledge from different information flows, and use this knowledge to improve the performance of the model on each local sub-task or global task. For this purpose, a multi-view shared learner, LSE is added to our TARSS-Net. The inputs of LSE are temporal-compressed embeddings (TCEs) obtained by temporal compression (TC) module, which consisted with a global temporal average-pooling layer and a 2D convolution layer with 1×1 kernels. Two shared 1×1 2D convolution layers are stacked to form the LSE, which will project the TCEs into a common latent space to get a uniform embedding. Since multi-view inputs are used, the 1 × 1 2D convolution layers are chosen instead of the fully connected layers for reducing the model parameters.Decoder. After encoder, the features obtained from TRAM and LSE could comprehensively consider single-view temporal relationship and multi-view relationship alignment, which will be fed into decoder. Only two branches for RD and RA views are designed in decoder, which is enough to complete target location. By the way, this is also the reason why existing RSS datasets is not labeled on AD view with groundtruth. Each of decoders consists of the combination of 2D deconvolutions and convolutions as in [21] .Figure S3 : The overall framework of TARSS-Net: details of the encoding branch for RD-view is given as an example.In general, Fig. S3 gives an intuitive illustration of the overall structure of our TARSS-Net. The encoding branch would take multi-view radar tensor (i.e., RD, AD, and RA) streams within several time frames as input, and generate the unified representation of these streams in a common latent space. Thus the encoding branch is used to learn representations and the decoding branch is used to interprete those representations. Besides each basic encoder, the proposed TRAM and ASPP modules aim to enhance the representation by capturing and measuring target-history relations and injecting multi-scale spatial information from each single view. Using these enhanced representations as input, the decoder would make cell-wise predictions on RD and RA views, respectively.The TRIC (temporal-relation-inception convolution) block proposed in TH-TRE ( § 3.1) can also be implemented with the 3D convolution. Firstly, let's recall the introduced implementation of TH-TRE:T H-T RE {X j } t j=t-τ = T RIC X t , {X i } t-1 i=t-τ ⊕ T Max (K 1 (X t )) , where T RIC X t , {X i } t-1 i=t-τ = K 2 K 1 (X t ) ⊕ D K 1 (X i ) t-1 i=t-τ . (S5)Where K 1 (•) and K 2 (•) are 2D convolution layers. The 3D-based implementation of TRIC can be conducted as follow:T RIC * X t , {X i } t-1 i=t-τ = K * 2 X * t ⊕ D X * i t-1 i=t-τwhereEQUATION)Where K * 1 (•) : R C×(τ +1)×H×W → R C×(τ +1)×H1×W1 and K * 2 (•) : R 2C×τ ×H1×W1 → R C×τ ×H2×W2 are two 3D convolution blocks with a kernel size of 1 in temporal dimension. Thus, the kernels are temporal-shared for each convolution layer. Then if 2D conv blocks of the same layer in Eq. S5 (the original) are shared, TRIC would be equivalent to its 3D version (TRIC * ). In this case, the numbers of parameters of TRIC and TRIC * are the same, i.e., (C ×1×3×3×C)+(2C ×1×3×3×C).However, when we review the present implementations of TRIC and TRIC * , we realize that they are all in a stacked form, i.e., K 1 /K * 1 is always nested in K 2 /K * 2 . It means that, the calculation of K 2 /K * 2 has to wait for the outcome of K 1 /K * 1 . To this end, a more parallel and elegant implementation of TRIC * comes up, and Fig. S4 intuitively describes this implementation. The key idea of this implementation is temporal cross reorganization for target-history feature pairs and temporal-shared 3D Conv-based relation encoding with 2 × 3 × 3 kernels and time stride of 2. In this case, the parameter scale are keeping the same with present TRIC * with 1 × 3 × 3 kernels, while enjoying a more parallel structure and less computation since there is no depth-concatenation and computation of the two Conv blocks are decoupled. More experiments and analysis will be conducted in the future work about these implementations.Three datasets have been used to valid the performance of TARSS-Net, including CARRADA [22] , CARRADA-RAC and KuRALs.CARRADA is a camera-radar synchronised dataset with large scale, which contains multi-view annotated radar recordings (RAD tensors) collected from a low-cost FMCW radar in various scenarios under different weathers conditions. There are 4 categories of objects: pedestrian, cyclist, vehicle and background; and multiple same/different types of moving objects could appear in a single time frame. The dimensions of RAD tensor are 256, 256 and 64, respectively. For fair comparisons, RSS-Net and RAMP-CNN were modified as the multi-view version, and all the SoTA methods including our TARSS-Nets were trained with the same setup.CARRADA-RAC is an improved version of CARRADA. As mentioned in § § 4.1, CARRADA is a camera-radar synchronised dataset. Its RD and RA annotation of were generated in a semi-automatic way [22] , in which the unreliable depth estimation of camera image and low angle resolution of FMCW radar reduce the labeling accuracy, especially in RA view. CARRADA-RAC further improves CARRADA by correcting its RA annotation accurate [35] . The training, validation and test subsets were split as in [21] .KuRALS is a large-scale self-collected real dataset with refined annotation. It is produced by a Ku-band continuous wave radar and includes various typical targets such as UAVs, airplanes, ships, and cars, which could support several radar sensing tasks including RSS, target tracking and so on. There are only RD tensors in KuRALS, so TMVA-Net, PKCIn-Net as well as TARSS-Net shown in Table 3 are modified to single view version. The complete dataset is forthcoming.The training setup for our proposed models and other compared SoTA networks is strictly consistent as follows:Input form: The raw 3D RAD tensor of each time frame was first compressed as 2D RA, AD and RD views with the input sizes of 256 × 256, 256 × 256 and 256 × 64. For the networks without explicitly considering the temporal information, e.g., RSS-Net [14] and MV-Net [21] , 2 historical frames (i.e., τ = 2) were added to form the input sequence. Besides, for the temporal information-based networks, e.g., RAMP-CNN [6] , TMVA-Net [21] , and our TARSS-Net, more historical frames were used: τ = 8 for RAMP-CNN, and τ = 4 for TMVA-Net and TARSS-Net.Running platform: The experimental platform of this work is RTX 3090 GPU with 24G memory. Both the single-view version and the multi-view version of TARSS-Net can be trained and tested on a single 3090 GPU.Hyper-parameters: All the models were trained with Adam optimizer [15] using the default setting of hyper-parameters, β 1 = 0.9, β 2 = 0.999 and ϵ = 1e -8. The initial learning rate was 1e -4, which decayed exponentially with the rate of 0.9 every 20 epochs. The training epochs and mini-batch size were 300 and 6, respectively.Evaluation metrics: For comprehensive comparisons, we evaluated our methods using cell-wise precision/recall, Intersection over Union (IoU), and Dice score. As the principle metrics, mean IoU (mIoU) and mean Dice (mDice) are calculated by category averaging. These two metrics are used to compare the performance of different methods on three validation datasets, as shown in Table 1 , Table 2 and Table 3 . In ablation experiments, as shown in Table 4 , Table 5 and Table 6 , precision and recall are also presented to comprehensively measure the effect of each design in TARSS-Net. All the reported results were obtained from the test subset.Since prediction results are generated on RD and RA views, AD view is actually embedded in the decoding branch. To further verify the effect of AD view in the multi-view RSS framework, we conduct an additional experiment on AD encoding branch (AD Enc.), both Baseline-B (replacing TRAM from TARSS-Net with emporal aggregation layers designed in TMVANet) and TARSS-Net_D (TRASS-Net with Depth-TRAP) were used for the experiment. Results in Table S1 clearly show that models can achieve better performance by using additional AD view. This performance improvements mainly come from the information compensation, i.e., AD view compensates RD with angle frequency response in angle domain and compensates RA with Doppler information. Such results also reveal that, taking full advantage of the information in the radar signals is indeed helpful for radar scene understanding. Hence from the perspective of information utilization, our method propose a pathway to joint analysis of multiple spatio-temporal domains of radar data. There are 4 categories of objects in CARRADA dataset, including: pedestrian, cyclist, vehicle and background. The intuitive comparison of class-wise performances is shown in Fig. S5 . TARSS-Net_D has the best overall RSS performance for most types of objects, especially for cyclists, while TARSS-Net_S performs better in the segmentation of pedestrians. RSS results are influenced by physical properties and motion states of different objects, which makes Depth-TRAP and Spatio-TRAP play their respective advantages. How to further improve TARSS-Net by combining object characteristics is also one of our future research directions.Some feature visualization of core process in TARSS-Net are shown in Fig. S6 . It confirms that the proposed novel temporal information learning paradigm, i.e. TRAM, is effective to make network learning pay attention to RSS target. Limitations. i) As can be seen in Table 1 , TARSS-Net performs worse than PKCIn-Net (which incorporates the advantages of classical radar detection theories), T-RODNet and TransRSS (which enhance spatial feature representation based on Transformer) in RA view where the temporal information is not obvious. ii) Deeply exploration on the learning principle and applicable scenarios of TARSS-Net_D and TARSS-Net_S is needed in future work. iii) TARSS-Net makes good trade-off between real-time performance and RSS effects. However, there is still room to improve its inference speed. These limitations also point to the improvement direction of TARSS-Net. Social Impacts. As for radar detection, TARSS-Net is applicable to various practical applications, such as perceptions for autonomous driving, UAV surveillance and marine monitoring. Furthermore, the proposed novel paradigm of temporal relationship learning is causal and can be exploited by more domains involving sequence data proceeding, including stock prediction, weather prediction, etc. However, inappropriate usage may lead to decreased reliability, potentially resulting in safety and other issues.Justification: For each theoretical result, this paper provides the full set of assumptions and a complete (and correct) proof.• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.This work focuses on exploiting temporal information in radar signals to enhance the representation capacity of multi-view RSS network. Firstly, the advantages and disadvantages of existing temporal modeling methods in RSS domain were deeply discussed, and on this basis, the design principles of RSS spatio-temporal encoding methods were introduced. Based on the principles, a flexible temporal-aware learning module, TRAM, and TARSS-Net based on TRAM is proposed, which follows the novle temporal learning paradigm, i.e. data-driven temporal information aggregation with learned target-history relations. Experiments fully verifies the superiority of TARSS-Net through SoTA methods comparison on three datasets, ablation experiments, performance under variation input time length, as well as its real-time performance. 


Decision: This paper receives an overall rating of 4/5/5/5 (average: 4.75). After the rebuttal, two positive reviewers were generally satisfied with the rebuttal and chose to maintain their ratings, while two other reviewers did not respond. The AC read the paper and all discussions. Although the technical contribution is not significant, it was well-prepared and improved the accuracy of radar-data segmentation. Additionally, the AC thinks the main issues raised by the reviewers have been addressed. So, the AC recommends accepting this paper, and urges the authors to add the supplementary contents during the rebuttal to the final paper.

Reviews: review: paper_summary: paper_summary: This paper proposed a temporal-aware framework, TARSS-Net, to enhance Radar semantic segmentation. The key idea is to propose a Temporal relation attentive module, TRAM (consists of Target-History Temporal Relation Encoding [TH-TRE] and Temporal Relation-Aware Pooling [TRAP] ), to capture the relations between Radar sequences.  Comparisons with baseline methods on three datasets (CARRADA, CARRADA-RAC and a self-collected KuRALS) show the advantages of the proposed method
strength_weakness: strength_weakness: -   Since there are many existing works to incorporate spatio-temporal information, the technical contribution is limited without the insights considering the data characteristic of Range-Angle-Doppler data.
- The performance increment is marginal, although various modules are designed.
- It lacks of runtime statics to evaluate the time consumption introduced by temporal design.
questions: questions: - How are the results when two frames are considered?
- The performance decreases after 6 frames and increases again from 9 frames. Is there any analysis of that?
- Can the authors provide to some feature visualization to confirm that whether the designed scheme payed attention to the object (or where to pay attention) in Radar sequences.
limitations: limitations: The authors appropriately discussed the limitations in the appendix.
paper_summary: paper_summary: This paper primarily introduces a network model called TARSS-Net, designed for the task of radar semantic segmentation. It effectively utilizes the temporal information of radar signals by introducing a novel temporal modeling mechanism, enhancing radar semantic segmentation performance. Specifically, the paper proposes a module called TRAM for temporal relationship learning between the target and historical frames, integrating it with other core components to construct the TARSS-Net model.
strength_weakness: strength_weakness: 1.This paper applies the time modeling to the field of radar semantic segmentation reasonably, but there are few innovations in the paper. 
2.The performance of time modeling approaches tends to degrade in crowded scenarios.
3.It is difficult to significantly improve the performance of RA-View under the time modeling method
4.The related works reviewed for the radar semantic segmentation task are not comprehensive.
5.The placement of illustrations within the main text is not particularly reasonable, making re-display in the Appendix an unsatisfactory choice.
questions: questions: 1.Can you provide a more detailed analysis of the computational complexity?
limitations: limitations: NA
paper_summary: paper_summary: In this work, the authors created a network called TARSS-Net for radar semantic segmentation. Compared to traditional methods, the authors emphasized the superiority of their approach in the clever utilization of historical information. Specifically, TARSS-Net incorporates a module called TRAM, which is designed similarly to the attention mechanism. This module first learns the relationship between historical frames and the current frame through the TH-TRE module, and then maps this learned relationship to hidden representations via TRAP. TRAM is specifically designed for radar semantic segmentation, ensuring that computational complexity remains manageable while achieving efficient and accurate radar semantic segmentation across three real radar segmentation datasets.
strength_weakness: strength_weakness: 629/5000
This paper may have the following potential shortcomings:

(1) Lack of Clear Hypotheses and Reasoning: The authors designed a novel temporal modeling paradigm for RSS and claimed its effectiveness in the context of RSS semantic segmentation. However, they seem to provide their viewpoint directly without offering clear hypotheses and reasoning about why this paradigm is effective for RSS semantic segmentation, which may confuse the readers.

(2) Insufficient Emphasis on the Design Motivation: The authors lack an emphasis on the motivation behind the model design. For instance, regarding the design of the TH-TRE module, the authors aim to use this module to encourage the model to focus more on high-dimensional information of the current time frame. However, Section 3.1 lacks an emphasis on this design motivation and focuses too much on formula statements. Given the considerable complexity of the designed method, it is crucial to introduce the design concept behind the module to the readers.

(3) Issues with Paper Formatting and Visualization: There are some issues with the paper's formatting and visualization. For example, the sequence of Figures 4 and 3 does not align well with typical reading habits, leading to some reading difficulty. Additionally, some elements in the figures are not drawn rigorously; for instance, the input of the TH-TRE module should be a 4D tensor, but Figure 3 depicts it more like a 3D tensor. For readers accustomed to visual aids, the authors' lack of rigorous visualization might lead to a misunderstanding of the model details.

(4) Potential Issues in the Experimental Section: There may be shortcomings in the experimental section, especially in comparison with the use of the Self Attention (SA) model in the temporal domain. Although the authors mention that applying SA in the temporal domain might lead to excessive computational consumption, the appendix shows that they have implemented a baseline model applying SA in the temporal domain based on ViT. Therefore, a more rigorous experiment should include this baseline model in the accuracy analysis (since accuracy analysis does not involve computational efficiency), as the authors have criticized this method in their analysis.
questions: questions: Some questions are as follows:

(1) What is the core innovation of this paper? Causal dilated convolutions also seem to align with the paradigm proposed by the authors. Compared to these simpler methods, what advantages does the method in this paper have? Why is it more advantageous?

(2) Is it meaningful to discuss real-time performance for RSS semantic segmentation? If RSS semantic segmentation in most scenarios demands higher accuracy rather than efficiency, then the criticism of other methods in terms of efficiency in the paper seems less significant. Can the authors provide specific examples of scenarios to analyze and explain this?
limitations: limitations: The authors adequately discuss the limitations.
paper_summary: paper_summary: This paper proposes a novel framework designed to integrate temporal information into radar-based semantic segmentation tasks. To achieve more effective temporal information integration, the framework introduces two key modules: the Target-History Temporal Relation Encoder (TH-TRE), which analyzes the relationships between different time frames, and the Temporal Relation Attentive Pooling (TRAP) module, which aggregates information along the temporal axis.
strength_weakness: strength_weakness: Please consult the question section for further information.
questions: questions: 1) Could the authors provide a comparison in terms of computational time?

2) Although the primary baseline, PKCIn-Net, also incorporates temporal information, the difference in performance between the proposed algorithm and PKCIn-Net appears to be minimal.

3) Additionally, the authors mention that this paper considers compression from both spatial-temporal and depth-temporal dimensions. Therefore, a pertinent question arises: would it be possible to first fuse spatial and depth information, and then subsequently fuse this combined data along the temporal axis?
limitations: limitations: Please consult the question section for further information.

Comments: title: Final reminder for LAST 3 Hrs of Discussion
Please take some time to review our new responses
Please take some time to review our new responses
We appreciate your respect for our hard work
We appreciate your respect for our hard work
Looking forward to your comments on our new reply!
Looking forward to your comments on our reply
Looking forward to your comments on our reply
Further response to [W1]
Replying to Official Comment by Reviewer awi8
comment: Dear Reviewers,

Sorry to bother you, but this is the last reminder from the authors. **We sincerely invite all reviewers to review and evaluate our responses**. We thank you for your hard work in the early stage and hope to get your short attention again.

Since **there are only three hours left for discussion**, we attach great importance to this submission and would like to buy you a little time at the end.

Warm regards,

9831 Authors
Dear Review WAmw,

Thank you for you hard work! The discussions we have had not only help to improve the quality of this paper but also show respect for our hard work. Thank you again. We have responded to your new concerns accordingly. Since the rebuttal is almost over, please take some time to review our new responses. **If you are satisfied with the replies, we especially appreciate you raising the evaluation score**. If there are any **new concerns**, we look forward to continuing the **in-depth discussion with you**.

Sincerely, 

9831 Authors.
Dear Review awi8,

Thank you for you hard work! The discussions we have had not only help to improve the quality of this paper but also show respect for our hard work. Thank you again. We have responded to your new concerns accordingly. Since the rebuttal is almost over, please take some time to review our new responses. **If you are satisfied with the replies, we especially appreciate you raising the evaluation score**. If there are any **new concerns**, we look forward to continuing the **in-depth discussion with you**.

Sincerely, 

9831 Authors.
Dear Reviewer kN5s,

We have spent a significant amount of time and effort analyzing your concerns and suggestions, and have provided detailed explanations and corresponding modifications. We believe that our response should be sufficient to address your concerns. We **sincerely hope that you would take the time to read our response**. If our response is **adequate**, we kindly ask you to give a **fair score upgrade**; if you still **have other concerns**,  looking forward to **further discussions with you**. Thank you for your contribution to improving the quality of our paper, and we also appreciate your respect for our hard work.

9831 Authors
Dear Reviewer MgaH,
We have spent a significant amount of time and effort analyzing your concerns and suggestions, and have provided detailed explanations and corresponding modifications. We believe that our response should be sufficient to address your concerns. We **sincerely hope that you would take the time to read our response**. If our response is **adequate**, we kindly ask you to give a **fair score upgrade**; if you still **have other concerns**,  looking forward to **further discussions with you**. Thank you for your contribution to improving the quality of our paper, and we also appreciate your respect for our hard work.

9831 Authors
We have updated responses to your new comments. Hope to get your approval on the reply and update the rating. We also look forward to more in-depth communication and discussion with you. Thank you again!
We have provided detailed responses to all of your questions. Hope to get your approval on the reply and update the rating. We also look forward to more in-depth communication and discussion with you. Thank you again!
We have provided detailed responses to all of your questions. Hope to get your approval on the reply and update the rating. We also look forward to more in-depth communication and discussion with you! Thank you again!
Dear Reviewer WAmw,
We appreciate your recognition of our response and further suggestion for modification. TransRSS indeed sets a good example in the radar detection background introduction, with a clear chain of thought. However, we did not choose this writing logic for the following two reasons:

1. **TARSS-Net has different methodology** from that of TransRSS, the essence of the problem in TARSS-Net's hand is the efficient temporal encoding of radar high-dimensional spatio-temporal tensors. In the initial version of TARSS-Net's introduction, we also began by *related background knowledge*, including the similarities and differences between radar devices and other visual devices, radar signal processing pipeline, and the characteristics of the radar data in hand. Then, we summarized *the development of radar target detection methods along a timeline*, ultimately highlighting the importance of temporal relationship modeling. However, unlike TransRSS, which combines Transformer and CNN in a way that is more easily understood and accepted by general AI researchers, such writing style would fail to allow readers to realize the limitations and challenges of existing temporal modeling methods in radar signal processing. That is, **given the existence of methods like 3DConv, RNNs, and Transformer, why is there a fundamental need for TARSS-Net?** After repeated discussions and revisions, we chose to quickly **highlight the current state of research in temporal relationship modeling in RSS** in the limited space of the Introduction, **point out the research gaps**, and then provide **a detailed summary of existing temporal modeling methods** in Sec. 2. We deeply analyze why existing methods are not suitable for RSS and **how to design temporal modeling methods suitable for the RSS field** by addressing these limitations.
2. Currently, **TARSS-Net has a higher research starting point**. While organizing this work, we found that excellent works like TMVA, PKC, and TransRSS *already possess the complete chain of thought you expect in their Introductions*. Therefore, we boldly omitted some common background knowledge that has been thoroughly discussed, allowing us to *take a higher perspective to discuss and analyze the RSS task from the entry point of temporal modeling mechanisms*. This represents new cognition and understanding not present in current research work in this field. We believe these contents can bring new insights and more inspiration to readers, which is also why we believe this work is suitable for the NeurIPS community. TARSS-Net, standing on the foundation of existing excellent work, brings fresh perspectives and cognition.

However, we must admit that since we placed more emphasis on temporal modeling, we had to omit some background information that does not affect the understanding of this paper under the constraints of the current limited space. This background information is included in the appendix. For example, in Sec. A From Radar Signals to Multi-View Representations, readers can understand the detailed process of millimeter-wave radar data processing in autonomous driving scenarios. In Sec. C Additional Descriptions of TARSS-Net, readers can gain a deeper understanding of the overall thought process of TARSS-Net in conjunction with the Methodology described in the manuscript. However, we agree with your suggestion. **Since additional page will be allowed in the camera-ready version, we promise to supplement the background introduction that cannot be added at this stage in the introduction, including the *characteristics of radar data* and *the challenges associated with processing it* you mentioned**, to better present the ***clear thought process*** of this paper. This will allow readers to easily grasp our intentions without having to read other materials or appendix as much as possible. Thank you for your contributions and thoughts to improve the quality of this paper. We sincerely express our respect to you and if you are satisfied with our new response to **[W1]**, we also look forward to your higher evaluation, so that more readers can see the higher quality TARSS-Net after the revision!
Thank you for your response and the detailed explanation of my questions. I believe addressing these issues is crucial for refining TARSS-Net. Regarding the authors' response to [W1. LACK OF CLEAR HYPOTHESES AND REASONING], as a reader, it is beneficial to understand the characteristics of the input data in specific scenarios and the challenges associated with processing it. A clear thought process—such as the analysis and reasoning behind the hypotheses proposed by the authors, followed by the methods tailored to specific scenarios—can help readers with a general AI research background better comprehend the intentions behind the authors' work. In this regard, one of the baseline methods in this paper, TransRSS, presents its ideas more effectively in the Introduction section.
First, we want to thank you for recognizing the contributions and innovations of our paper. Then thanks for your more detailed explanation about **[W2 PERFORMANCE DEGRADES IN CROWDED SCENARIOS]**. Your mention of *multiple targets* has clarified your definition of *crowded scenarios*. The common multi-target situations in radars are already included in the CARRADA and our self-collected KuRALS datasets (e.g., the simultaneous presence of pedestrians and cars, multiple drones/UAVs or multiple ships). For the crowded scenarios you mentioned, we believe the closest examples are **drone swarm detection** and **bird flock detection**:

- Taking the **drone swarm detection** scenario as an example. Typically, the radar will first perform a wide-area search, i.e., in the scanning mode it will emit coherent/incoherent pulse trains in each direction. The use of coherent pulse trains is more common. In each direction, these coherent pulse trains will form what we call RD representations. Considering the flight safety distance of the drones themselves, **the number of targets that can be covered in the same direction (i.e., one RD frame) will be much smaller than the entire swarm size**. In the end, it is difficult for single RD frame to reflect the crowded scenarios that appear in imaging signals such as camera images and SAR data. Subsequently, radar gets in the tracking mode, the system will form multiple tracking channels with smaller detection range, **making each channel cover only the spatial range of one target as much as possible**. After accumulating tracks over multiple RD frames, it will perform target identification or other more refined perception tasks. Therefore, whether in the scanning or the tracking mode, **it is difficult to obtain RD frame which can reflect so called crowded scenario**. Moreover, collecting such radar data for drone swarms is very challenging, requiring the cooperation of very professional flight control technicians and radar technicians to successfully collect this type of data.
- In the **bird flock detection** scenario, the flight distance of birds (about 1~2 meters during the migration of large birds and tens of centimeters when small birds are foraging) is often lower than the radar's range resolution. For example, the Ku-band radar we use for collecting KuRALS dataset has a radial range resolution of about 3 meters. Therefore, **the bird flock targets will all appear as a single mass target on one RD frame**. At this point, *the crowded scenario in vision collapses into a connected domain in the radar perspective*.
- Finally, let's analyze a more common crowded case, namely **heavy traffic**. Unlike vision, the form of crowded traffic scenarios in RDs cannot satisfy what you mentioned as crowded scenarios. This is because the R axis means radial distance, and D axis means Doppler or velocity. In heavy traffic, the speeds of the moving targets are similar, which results in a line pattern in parallel with R axis of RD frame. Such situations are included in the dataset involved in our experiments, where TARSS-Net can demonstrate its effectiveness.

In summary, the *crowded scenarios* you refered to may appear more frequently in camera or SAR imaging radar. However, in the pulse Doppler or continuous wave radar discussed in this paper, crowded scenarios are generally diluted across different channels at signal processing level or collapsed into a single "target" due to the radar's physical characteristics. Therefore, **visual crowding cannot be directly mapped to the radar scenarios discussed in this paper**. The crowded scenarios you mentioned exceed the scope of signal processing algorithms and are a systematic issue. Radar designers will consider multiple aspects to design corresponding solutions, not just relying on signal processing algorithms.

TARSS-Net proposes a general method for improving RSS performance. The datasets used in this paper cover typical radar application fields, such as autonomous driving and low-altitude surveillance. The experimental results also demonstrate the effectiveness of TARSS-Net. Hence the impact of crowded scenarios on the performance of radar temporal modeling methods is not within the scope of this paper. Nonetheless, thanks for your valuable concern, which motivates us to do more inspiring discussions and points us in a new research direction of this field. We will specifically consider this situation in subsequent studies. The TARSS-Net proposed in this paper has already demonstrated its effectiveness in improving the performance of general radar detection scenarios. We are pleased that this has been recognized by you and other reviewers. If our response can address your concerns, we hope you can further provide a higher rating to inspire more peers and further improve radar signal processing capabilities.
Thank you for your response. I appreciate your further explanation of the TARRS-Net and the additional analysis of computational complexity. It is noteworthy that Crowded Scenarios refer to situations with a higher number of objects, and typically, time modeling may experience performance degradation in such scenarios. 

Overall, the authors have addressed some of the issues raised during the review process, reinforcing the significance of TARSS-Net as an innovative approach in the field of radar signal processing. I look forward to seeing the proposed improvements, and I will maintain my current rating.