###### Abstract
In addition, when applying the Viterbi and MAP algorithms, it is essential to accurately estimate the coefficients of the HDS channel. Thus, we used the method in [19] to compensate and predict the coefficients of the HDS channel. This method is inspired by the generalized partial response (GPR) target from [20]. In addition, we can use the serial or parallel structures in [21,22] to split the 2D interference into 1D interference, which is suitable for the Viterbi and MAP algorithms. Recently, interference estimators were introduced by [PERSON] and [PERSON] in [11] to predict vertical interference (VI) and horizontal interference (HI). Then, we can remove these interferences from the received signal to obtain the signal with 1D interference. Finally, we can apply deep learning (DL) models to the interference estimator to improve the performance of conventional estimator [11], utilizing the Viterbi algorithm to find out the interference. In [23], the authors proposed the interference estimator using the recurrent neural network (RNN) to improve the accuracy of estimation. Beyond the Viterbi algorithm, other detection methods have been derived from the BCJR algorithm. One such method is the iterative row-column soft-decision feedback algorithm (IRCSDFA), which was proposed to mitigate 2D interference but is computationally intensive [24]. To address this issue, [PERSON] et al. proposed a lower-complexity variant using a Gaussian approximation (IRCSDFA-GA). This version reduces computational complexity but is still more demanding than the Viterbi algorithm (VA) [25]. In this paper, a new scheme for serial MAP detection is introduced. In the proposed model, instead of using a parallel structure to exploit extrinsic information, we utilized the interference estimator in [11] to exploit the extrinsic information for serial MAP detection. Firstly, the received signal is split into two parts. One was fed to the interference estimator to extract the interference symbols and convert into the extrinsic information for serial MAP detection. To predict the probability of the six-level signals and mitigate HI, another received signal and the extrinsic information were processed through the horizontal MAP detection. To convert the interference into extrinsic information, we introduced the parameter \(\alpha\) as the ratio of extracting the information into extrinsic information. Then, the probability of the six-level signal was detected by vertical MAP detection to remove VI and estimate the probability of the original signal. These probabilities can be used to create the iterative algorithm between the horizontal and vertical MAP detections. Finally, the probability of the original signal was compared to a certain threshold to obtain the original user data. The results from the simulation clearly indicate that our proposed model significantly improves the bit-error rate (BER) performance of HDS systems compared to previous studies. Our contributions are as follows:  
* Proposed the method to convert the estimated interference into extrinsic information. * Found out the optimized value of \(\alpha\) (the ratio of extracting the interference into extrinsic information). * Analyzed and explained the combination of the interference estimator and the serial MAP detection. * Presented comprehensive simulation results to validate the effectiveness of the proposed model. The results demonstrate significant improvements in BER performance compared to existing approaches, confirming the robustness of our method. The rest of this paper is structured as follows: Section 2 provides a brief introduction to the background of the GPR target, interference estimator, and serial MAP detection. Section 3 provides a detailed discussion of the channel modeling for HDS systems, along with a comprehensive explanation of the proposed scheme. In Section 4, we present and discuss the simulation results. Lastly, Section 5 concludes the paper.