## 3 Proposed Model
### Serial MAP Detection  
With the parameters \(r\), \(p\), \(l_{1}\), and \(l_{2}\), we can use the serial detection in Figure 2 for the received signal. The MAP algorithm is applied to horizontal detection and vertical detection as follows. The signal \(z[j,k]\) is similar to the signal influenced by the vertical interference \([p\;l_{2}\;p]^{T}\) and the horizontal interference \([r\;l_{1}\;r]\). Thus, serial detection consists of horizontal detection followed by vertical detection. After applying horizontal detection to \(z[j,k]\), we obtain the signal \(b[j,k]\), which is similar to \(a[j,k]\bigotimes[p\;l_{2}\;p]^{T}\). Since \(b[j,k]\) is the output of horizontal detection, it no longer contains the horizontal interference factor \(r\) but retains the vertical interference factor \(p\). Given \(a[j,k]\in\{-1;1\}\), the possible values of \(a[j,k]\bigotimes[p\;l_{2}\;p]^{T}\) are \(\{-l_{2}-2p;\,-l_{2};\,-l_{2}+2p;\,l_{2}-2p;\,l_{2};\,l_{2}+2p\}\). After horizontal detection, only vertical interference remains in the received signal. Similarly, vertical detection can then be applied to \(b[j,k]\) to recover \(a[j,k]\), which closely approximates the original signal \(a[j,k]\). Figure 2: Serial detection. * Horizontal detection The signal \(b[j,k]\) can be estimated as below without noise in channel. \[b[j,k]=a[j,k]\otimes\left[pl_{2}\quad l_{1}l_{2}\quad pl_{2}\right]^{T},\] (35) where \(\bigotimes\) is the convolution operator. Due to \(a[j,k]\) being modulated and represented as \(\{-1/1\}\), the signal \(b[j,k]\in\{-2p-l_{1};\,-l_{1};\,2p-l_{1};\,-2p+l_{1};\,l_{1};\,2p+l_{1}\}\). We assigned the values of \(b[j,k]\) as the symbols \(\{R_{1}=-2p-l_{1};\,R_{2}=-l_{1};\,R_{3}=2p-l_{1};\,R_{4}=-2p+l_{1};\,R_{5}=l _{1};\,R_{6}=2p+l_{1}\}\). With six symbols, horizontal detection was performed using a trellis with thirty-six states, each having six branches. The conditional probability \(P(R_{i}\,|\,\mathbf{z})\) for the six symbols was calculated using the equation below. \[ \begin{split} P(\,R_{i}|\,\mathbf{z})&=\sum\limits _{R_{i}}P(s^{\prime},s|\,\mathbf{z})=\sum\limits_{S_{i}}\frac{P(s^{\prime},s| \,\mathbf{z})}{P(\mathbf{z})}\\ &=\sum\limits_{S_{i}}\frac{P(s^{\prime},\mathbf{z}_{<k})P(\mathbf{ z}_{k},s|\,s^{\prime})P(\mathbf{z}_{>k}|\,s)}{P(\mathbf{z})}\\ &=\sum\limits_{S_{i}}\frac{\alpha_{k-1}(s^{\prime})\gamma_{k}(s^{ \prime},s)\beta_{k}(s)}{P(\mathbf{z})}\\ \text{with }i=1,\;2,\;\ldots,\;6\end{split} \] (36) where \(s^{\prime}\), \(s\), and \(\mathbf{z}\) are the previous state, current state, and received symbol of the MAP algorithm, respectively. \(\alpha_{k-1}(s^{\prime})=P(s^{\prime},\,\mathbf{z}_{<k})\) is the joint probability of the previous state \(s^{\prime}\) and sequence received in the past \(\mathbf{z}_{<k}\). \(\gamma_{k}(s^{\prime},s)=P(\mathbf{z}_{k},s\,|\,s^{\prime})\) represents the conditional probability of the current state \(s\) and the current received symbol \(\mathbf{z}_{k}\) given the previous state \(s^{\prime}\), and \(\beta_{k}(s)=P(\mathbf{z}_{>k}\,|\,s)\) is the conditional probability of the future received sequence \(\mathbf{z}_{>k}\) under the current state \(s\). Next, we calculate the branch transition probability as below. \[\gamma_{k}\big{(}s^{\prime},s\big{)}=P(z|Z_{i})P(R_{i})=\exp\left(-\frac{(z-Z_ {i})^{2}}{2\sigma^{2}}\right)P_{e}(R_{i}),\] (37) where z represents the value of \(z[j,k]\) and \(Z_{i}\) represents the discrete value of z in the noiseless scenario (216 symbols on the trellis). The probability distribution \(P(z\,|\,Z_{i})\) is modeled using a Gaussian function. Additionally, \(P_{e}(R_{i})\) is the extrinsic information obtained from vertical detection and used to create the iterative algorithm in serial MAP detection. * Vertical detection In vertical detection, the probability of values in signal \(a[j,k]\) is estimated. As a result, vertical detection employed the trellis with 4 states, where each state has 2 branches. The branch transition probability \(\gamma_{k}(s^{\prime},s)\) is estimated as follows.