# Kernel Point Convolution LSTM Networks for Radar Point Cloud Segmentation
## 3 Methodology
### Kplstm  
From an architecture point of view, the KPConv-CLSTM comes with the drawback that the time dependency is only applied on the global feature of the point cloud in the latent space. A direct application of the LSTM cells to the input points would not be feasible  
Figure 3: High-level structure of our KPConv model with ConvLSTM cells in the latent space. The blue color shows layers that consider time dependencies.  
Figure 2: Processing of the center point \(x\) with \(d_{out}\) kernel point filters. Based on [8].  
due to the irregular structure and size of the input point cloud over several time steps. [PERSON] [21] associates points of the current radar point cloud with neighboring points from a memory point cloud by using the grouping method of PointNet++, using only the current point cloud as center points. Similar to [16], we propose a new variant of the LSTM cell in such a way that it can process and associate point cloud data directly. Instead of pointnet-based processing, we use KPConv kernel gates for the KPLSTM cell. To align the dimension of the past and present point clouds, we explore different sampling strategies from nearest neighbor sampling to learnable sampling via the KPConv weights. For the use case of the paper, the nearest neighbor sampling proved most reliant. Figure 4 shows a comparison of a standard LSTM cell and the KPLSTM cell.  
The inputs to the LSTM cells are the old cell state \(C_{t-1}\), the old hidden state \(H_{t-1}\) and the current point cloud \(X_{t}\). The output of the cells are the current cell state \(C_{t}\) and the current hidden state \(H_{t}\). For the input data, the old hidden state and current point cloud are concatenated. This structure is the same as in the standard LSTM cell. Instead of fully connected layers, KPConv operations \(KP_{x}\) are used to model the input gate \(I_{t}\), output gate \(O_{t}\), forget gate \(F_{t}\) and internal cell state \(\tilde{C}_{t}\). The subscript \(x\) denotes the respective weights. The \(KP_{x}\) layers perform feature generation and point cloud association of the old and current coordinates at the same time. Additionally, the old cell state \(C_{t-1}\) needs to be mapped to the new point coordinates. As we do not want to apply the features of the new point cloud but only propagate the old features to new coordinates, we take the current point coordinates without the point features \(X^{\prime}_{t}\) and use them as the center points for the  
Figure 4: Comparison of standard LSTM cell and KPLSTM cell.  
nearest neighbor sampling \(S_{NN}\) to obtain the intermediate cell state \(\hat{\mathcal{C}}_{t-1}\). The remaining structure is analogous to the standard LSTM cell. The symbol \(\circ\) denotes the element-wise product. We remove old features from the cell state via the forget gate and add new ones via the input gate. The equations describing the KPLSTM cell are as follows:  
\[\hat{\mathcal{C}}_{t-1} =S_{NN}\big{(}X_{t}^{\prime},C_{t-1}\big{)}, \tag{1}\] \[I_{t} =\sigma(KP_{i}(X_{t},H_{t-1})),\] (2) \[F_{t} =\sigma(KP_{f}(X_{t},H_{t-1})),\] (3) \[O_{t} =\sigma(KP_{o}(X_{t},H_{t-1})),\] (4) \[\hat{\mathcal{C}}_{t} =\tanh(KP_{\mathcal{C}}(X_{t},H_{t-1})),\] (5) \[C_{t} =F_{t}\circ\hat{\mathcal{C}}_{t-1}+I_{t}\circ\hat{\mathcal{C}}_{t},\] (6) \[H_{t} =O_{t}\circ\tanh\big{(}C_{t}\big{)} \tag{7}\]  
Figure 5 shows the overall KPLSTM network architecture. The encoding is performed in two consecutive blocks. Each block comprises a KPConv layer to downsample the point cloud, a KPLSTM cell and another KPConv layer for feature generation. The upsampling is done in the same manner as in the KPConv-CLSTM. It is worth mentioning that the time dependency is already introduced in the network encoding layers, making an intermediate global feature LSTM model obsolete. With this, we capture local structure over time instead of only being able to keep track of the features globally.