## 3 MHSA-1D CNN-Bi-LSTM Interference Cancellation Scheme
### Multi-Head Self-Attention  
The MHSA mechanism is a variant of the self-attention mechanism, designed to enhance the expressive and generalization abilities of the model. Multiple attention computations are performed in parallel on the signal sequence, organized by a \"head\" structure. If there are \(h\) \"head\" structures, Q, K, and V undergo linear transformations and are allocated to each head, with \(1/h\) of the original dimension during the initial modeling stage. Subsequently, each \"head\" independently calculates attention using its allocated data. The computed results are then concatenated along a specific dimension and transformed through a linear operation to get the final attention distribution. The process can be expressed as in the following equations [22]:  
\[head_{i}=Attention(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}) \tag{8}\]  
\[MultiHead(Q,K,V)=Concat(head_{1},head_{2},\dots,head_{h})W^{O} \tag{9}\]  
where \(W_{i}^{Q}\), \(W_{i}^{K}\), \(W_{i}^{V}\), and \(W_{o}\) are learnable linear transformation parameter matrices. The two-head self-attention mechanism is depicted in Figure 4.  
\begin{table}
\begin{tabular}{l} \hline
**Algorithm** **Bi-LSTM** \\ \hline
**Start** \\ Input: Input Sequence \(X=[x_{1},x_{2},\dots,x_{T}]\), \(T\) is the length of the sequence. \\ Parameters: LSTM cell parameters \((W,U,b)\), \(W,U\) is weights and \(b\) is biases. \\ Output: Output Sequence \(H=[h_{1},h_{2},\dots,h_{T}]\) \\
**Step 1: Initialize hidden and cell states.** \\ \(h_{forward}[0]=0\), \\ \(c_{forward}[0]=0\), \\ \(h_{backward}[0]=0\), \\ \(c_{backward}[0]=0\). \\
**Step 2: Forward propagation.** \\ \(for\)\(t=1\) to \(T\) do, Loop statement that iterates from \(1\) to \(T\). \\ Calculate the output of forward LSTM: \\ \(i_{forward}[t]=\sigma(W_{i}*x_{t}+U_{i}*h_{forward}[t-1]+b_{i})\), \\ \(f_{forward}[t]=\sigma(W_{i}*x_{t}+U_{i}*h_{forward}[t-1]+b_{f})\), \\ \(o_{forward}[t]=\sigma(W_{o}*x_{t}+U_{o}*h_{forward}[t-1]+b_{g})\), \\ \(g_{forward}[t]=\tanh(W_{g}*x_{t}+U_{g}*h_{forward}[t-1]+b_{g})\), \\ \(c_{forward}[t]=f_{forward}[t]*c_{forward}[t-1]+i_{forward}[t]*g_{forward}[t]\), \\ \(h_{forward}[t]=o_{forward}[t]*\tanh(c_{forward}[t])\). \\
**Step 3: Backward propagation.** \\ for \(t=T\) to \(1\) do, Loop statement that iterates from \(T\) to \(1\). \\ Calculate the output of backward LSTM: \\ \(i_{backward}[t]=\sigma(W_{i}*x_{t}+U_{i}*h_{backward}[t+1]+b_{i})\), \\ \(f_{backward}[t]=\sigma(W_{i}*x_{t}+U_{i}*h_{backward}[t+1]+b_{f})\), \\ \(o_{backward}[t]=\sigma(W_{o}*x_{t}+U_{o}*h_{backward}[t+1]+b_{o})\), \\ \(S_{backward}[t]=\tanh(W_{g}*x_{t}+U_{g}*h_{backward}[t+1]+b_{g})\), \\ \(c_{backward}[t]=f_{backward}[t]*c_{backward}[t+1]+i_{backward}[t]*g_{backward}[t]\), \\ \(h_{backward}[t]=o_{backward}[t]*\tanh(c_{backward}[t])\). \\
**Step 4: Combine forward and backward outputs.** \\ for \(t=1\) to \(T\) do, \\ \(h[t]=concat(h_{forward}[t],h_{backward}[t])\), Splice forward and backward hidden states. \\ return to \(H=[h_{1},h_{2},\dots,h_{T}]\). \\
**End** \\ \hline \end{tabular}
\end{table}
Table 1: Bi-LSTM computational procedure.  
The distinction between the MHSA mechanism and the self-attention mechanism lies in the fact that the latter only necessitates a single \(W^{Q}\), \(W^{K}\), and \(W^{V}\) for computing Q, K, and V, whereas the former requires an additional computation of computed Q, K, and V. Taking \(q^{i}\) as an example, \(q^{i,1}=W^{q,1}q^{i}\), \(q^{i,2}=W^{q,2}q^{i}\) is initially computed, followed by the computation of \(b^{i,1}\) using \(q^{i,1}\), \(k^{i,1}\), \(v^{i,1}\) and \(q^{i,1}\), \(k^{j,1}\), \(v^{j,1}\). Subsequently, \(b^{i,2}\) is computed using \(q^{i,2}\), \(k^{i,2}\), \(v^{i,2}\) and \(q^{i,2}\), \(k^{i,2}\), \(v^{i,2}\). Then, \(b^{i,1}\), \(b^{i,2}\) are concatenated to obtain \(b^{i}\) through matrix multiplication with \(W^{O}\). Similarly, \(b^{i}\) can be obtained following this procedure, thereby completing the computation of two-head self-attention mechanism [23].