## 3 Architecture of the Proposed HDL Detector
Additionally, the LSTM has two states, the hidden state \(h_{t-1}\) and cell state \(C_{t-1}\). The cell state functions as memory to retain information extrapolated from previous inputs, while the hidden state is used to compute the output. Here, \(t\) represents the time instant and \(B_{t}\) is the current input. The gates function updates the cell state, enabling the Bi-LSTM cell to add or remove data at each time step. The forget gate and input gate control the amount of cell state that needs to be updated and reset, respectively [44]. The functions of the forget, input, output, and candidate gates in the Bi-LSTM model are expressed as follows:  
\[Fo_{t}=\sigma_{\psi}(W_{Fo}B_{t}+T_{Fo}h_{t-1}+b_{Fo}), \tag{8}\]  
\[In_{t}=\sigma_{\psi}(W_{In}B_{t}+T_{In}h_{t-1}+b_{In}), \tag{9}\]  
\[Out_{t}=\sigma_{\psi}(W_{Cu}B_{t}+T_{Cu}h_{t-1}+b_{Cu}), \tag{10}\]  
\[Ca_{t}=\sigma_{tanh}(W_{Ca}B_{t}+T_{Ca}h_{t-1}+b_{Ca}), \tag{11}\]  
\[\mathbf{\hat{y}}=\frac{y\cdot h}{|h|^{2}+\sigma^{2}} \tag{5}\]0 The internal configuration of an LSTM cell. where \(B_{t}\) denotes the input vector of the Bi-LSTM at iteration \(t\), the sigmoid activation function is represented by \(\sigma_{\Psi}\), the terms \(F_{0t}\), \(Int_{t}\), \(Out_{t}\), and \(Ca_{t}\) correspond to the respective activation vectors of the forget, input, output, and candidate gates at iteration \(t\), the vector \(h_{t-1}\) represents the hidden state from the previous iteration \(t-1\), and the weight matrices \(W_{Fo},W_{In},W_{Ou}\), and \(W_{Ca}\) are associated with the forget, input, output, and candidate gates, respectively, with regard to the input \(B_{t}\). Similarly, the weight matrices \(T_{Fo}\), \(T_{In}\), \(T_{Ou}\), and \(T_{Ca}\) are tied to the previous hidden state \(h_{t-1}\). Finally, the bias vectors for the forget, input, output, and candidate gates are represented by \(b_{Fo},b_{In},b_{Ou}\), and \(b_{Ca}\), respectively. The Bi-LSTM model updates the previous cell state \(c_{t-1}\) to the current cell state according to the following equation:  
\[\mathbf{\hat{y}}=\frac{y\cdot h}{|h|^{2}+\sigma^{2}} \tag{5}\]1  
where \(\mathbf{D}_{i}\) represents the true values, \(\mathbf{D}_{o}\) represents the predicted values, and \(n\) is the number of samples. Adaptive moment estimation (Adam) is a DL optimization technique that combines the strengths of both RMSprop and momentum. By adjusting the learning rate for each parameter, Adam efficiently handles sparse gradients and noisy data, making it well suited for complex models and large datasets [48]. Adam is readily implementable in a number of commercial DL platforms, including Keras 2.10.0 and TensorFlow 2.10.0. Additionally, we evaluate the performance of the RMSProp, Adagrad, and Gradient Descent optimizers for comparison. Figures 4 and 5 illustrate the training loss of the proposed model for the different equalizers and modulation schemes, respectively. We evaluate the training loss across each epoch to monitor the model's performance over time. As observed in Figure 4, the proposed model demonstrates strong training loss performance across all equalizers. While the DFE equalizer exhibits slightly higher loss, both the ZF and MMSE equalizers achieve very low and nearly identical training loss levels. Figure 5 illustrates the training loss for different modulation schemes under various data setups, specifically for \((4,1,4)\), \((8,2,8)\), and \((8,4,16)\) with the ZF equalizer. Notably, this training loss is estimated at 15 dB training SNR with a batch size of 1000 and learning rate of 0.005. As can be observed, the training loss tends to increase with higher modulation orders and data setup. As the model's performance depends significantly on the SNR level chosen for training, denoted by \(\mathcal{T}_{\text{train}}\); thus, careful selection is crucial in order to optimize HDL's effectiveness. Specifically, the model trained at this SNR level must perform reliably across other relevant SNR values, making it essential to select an optimal \(\mathcal{T}_{\text{train}}\).