# A Nonlinear Convolutional Neural Network-Based Equalizer for Holographic Data Storage Systems
## 3 Proposed Model
### Proposed Nonlinear Equalizer  
In the conventional linear equalizer (Section 2), the 2D received signal is convoluted with only one layer (\(5\times 5\) matrix \(\mathbf{F}\)) from the CNN perspective. To improve the conventional equalizer, we propose a CNN structure that extends many 2D convolution layers and creates a nonlinear equalizer. CNNs are frequently proposed in the image processing field to handle 2D signals. Therefore, in the HDS channel, the CNN is suitable for the 2D received signal on the project screen. Figure 3 shows the structure of the nonlinear equalizers. The proposed system model, based on a nonlinear CNN equalizer, is shown in Figure 4.  
In our proposed model, user signal \(u[k]\) is modulated into a 2D signal \(s[j,k]\) to make it suitable for the medium of the HDS system. Signal \(s[j,k]\) is distorted by the 2D ISI and electric noise \(w[j,k]\). Therefore, received signal \(r[j,k]\) is passed through the nonlinear equalizer (CNN) to reduce the 2D ISI and electric noise. Because of using the serial target form, signal \(z[j,k]\) can be detected using serial detection, as in [19]. Signal \(z[j,k]\) is passed through the horizontal detector to create signal \(b[j,k]\). Signal \(b[j,k]\) is then passed through the vertical detector to restore modulated signal \(s[j,k]\). Finally, user data \(u[k]\) are recovered by demodulating signal \(s[j,k]\). To estimate the BER, we compare and count the difference in bits between \(u[k]\) and \(\hat{u}[k]\). In this study, we tested eight CNN structures for a nonlinear equalizer, as illustrated in Figure 5. They were grouped into two types: straight and branch structures. Straight structures are typically used in conventional CNNs. Subsequently, the branch structure of the CNN was developed to reduce the vanishing gradient phenomenon (as explained in Section 3.3). To train the CNN model, we implemented the following steps:  
Figure 4: Diagram of the proposed model. Figure 3: Nonlinear structure for the CNN-based equalizer. Use MMSE algorithm, as specified in Section 2, to estimate conventional equalizer \(\mathbf{F}\) and target \(\mathbf{T}\). * Collect signals \(d[j,k]\) and \(r[j,k]\) when estimating \(\mathbf{F}\) and \(\mathbf{T}\). * Use signal \(r[j,k]\) as the input and signal \(d[j,k]\) as the label for the CNN model in the training process. * Replace equalizer \(\mathbf{F}\) with the trained CNN model. In addition, we used the following MSE function as the loss function for the CNN model: \[loss=\frac{1}{2\alpha\beta}\sum_{j=1}^{\alpha}\sum_{k=1}^{\beta}(r[j,k]-d[j, k])^{2}\] (16)  
Here, \(\alpha\) and \(\beta\) represent the total number of the rows and columns of signal \(y[j,k]\), respectively. Figure 5: Cont. In Figure 5, the \"Conv\" layers have a 5 \(\times\) 5 kernel and a stride of 2 \(\times\) 2. In addition, the \"Avgpool2D\" layers are subjected to average pooling with a size of 2 \(\times\) 2 and a stride of 2 \(\times\) 2. The \"Batchnorm\" layers used the below equation to normalize the data. \[\hat{x}=\frac{x-\mu_{x}}{\sigma_{x}} \tag{17}\]  
where \(x\) is the input of \"Batchnorm\" layer and \(\hat{x}\) is the output of \"Batchnorm\" layer. \(\mu_{x}\) is the mean of \(x\), and \(\sigma_{x}\) is standard deviation of \(x\). In addition, the ReLU function is presented in (18). \[f(x)= \begin{cases}x,&x>0\\ 0,&x\leq 0\end{cases}  \tag{18}\]  
In Figure 5b, to easily present the number of filters in \"Conv\" layers, we used the vector type for the number of filters as [Conv_1 Conv_2 Conv_3...]. Structure 3: [16 32 32]. Structure 4: [7 10 12 15 10 7]. Structure 5: [5 10 12 15 10 7 3]. Structure 6: [5 10 12 15 10 7 3]. Structure 7: [5 10 12 15 10 7 3]. Structure 8: [5 10 12 15 10 7 3]. Figure 5: Structures of the CNN: (**a**) straight structures and (**b**) branch structures. Based on the theory model, we can see that the CNN structure converts the received signal nonlinearly into the desired signal with a higher accuracy than the conventional equalizer structure.