## 4 Simulation and Results
### Third Proposed Model  
With the second model, we can achieve the desired results in theory. However, given the survey to determine the optimal parameter \(\alpha\), it is cumbersome. Therefore, we propose the third model, illustrated in Figure 16.  
Figure 16: The estimated noise power compensation model.  
Figure 15: The BER performance of the second model.  
This model uses the feedback from the signal \(\hat{a}[j,k]\) to estimate the interfered signal \(\hat{y}[j,k]\) without any random noise. We know that the received signal \(y[j,k]\) includes the 2D interfered signal \(\hat{y}[j,k]\) along with noise. Therefore, the received signal is subtracted by \(\hat{y}[j,k]\) to obtain the noise signal \(\hat{e}_{p}[j,k]\). Then, the parameter \(\alpha\) is given by the square root of the variance of the noise signal \(\hat{e}_{p}[j,k]\) and multiplied by the signal \(\hat{e}[j,k]\). The results of the third model are shown in Figure 17.  
For the third model, we trained the neural network at an SNR of 11 dB and used one to three hidden layers. To clarify the effectiveness of deep learning, we increase the training data. We use 5,000 random samples to train for this model. Although the complexity of the third model is higher than that of the second model, we do not need to conduct a survey to determine the optimal value of \(\alpha\) and do not spend considerable time on the training process. Moreover, this model achieves a performance similar to the second model.  
We also compared our proposed method with the BCJR algorithm, which is represented by the IRCSDF-GA algorithm in [21]. In [21], the authors also showed that the results in [20; 21] were similar; however, for simplicity, we implemented the algorithm in [21]. In theory, the MAP algorithm achieves a better performance than the ML algorithm. However, in [20; 21], the MAP algorithm is not suitable for 2D interference; therefore, it cannot achieve the theoretical results. For complexity, the IRCSDF-GA's trellis includes four states with two input branches in each state. The IRCSDF-GA algorithm has two trellises for rows and columns, respectively. In each trellis, the computation consists of determining the branch, forward, and backward metrics. Therefore, it is evident that the IRCSDF-GA is more complicated than the serial detection model. In the serial detection model, the inner detection trellis included four states and six input branches, and the outer detection trellis included four states and two input branches. Both the inner and outer detections' computations need only to find the branch metric.  
Furthermore, the ITI and ISI are equal in [20], whereas ITI is more severe than ISI in this study. Therefore, there is a gain difference between this study and [20]. In addition,  
Figure 17: The BER performance of the third model.  
in [20; 21], the authors designed the IRCSDF algorithm with a known channel parameter, and this is not appropriate for a real situation where we cannot explicitly identify the channel.  
In addition, we also compare our proposal with the ConvNet detector in [29]. However, the ConvNet detector cannot achieve high performance, because the ConvNet detector is designed for collaborating with the low-density parity code (LDPC). Therefore, the ConvNet detector achieves low BER performance when we use it alone.  
From the results of the second and third models, we can see that the parameter \(\alpha\) helps the neural network adapt to the environment when the SNR is high. Also, the performance is almost the same even if the number of hidden layers is changed.