# A Nonlinear Convolutional Neural Network-Based Equalizer for Holographic Data Storage Systems
## 2 Background  
In HDS systems, an equalizer is set up at the channel output, as shown in Figure 2.  
We define the equalizer as matrix \(\mathbf{F}\) as follows:  
\[\mathbf{F}=\begin{bmatrix}f_{-2,-2}&f_{-2,-1}&f_{-2,0}&f_{-2,1}&f_{-2,2}\\ f_{-1,-2}&f_{-1,-1}&f_{-1,0}&f_{-1,1}&f_{-1,2}\\ f_{0,-2}&f_{0,-1}&f_{0,0}&f_{0,1}&f_{0,2}\\ f_{1,-2}&f_{1,-1}&f_{1,0}&f_{1,1}&f_{1,2}\\ f_{2,-2}&f_{2,-1}&f_{2,0}&f_{2,1}&f_{2,2}\end{bmatrix} \tag{1}\]  
Signal \(z[j,k]\) is calculated as follows:  
\[z[j,k]=\sum_{n=-2}^{2}\sum_{m=-2}^{2}f_{n,m}r[j-n,k-m] \tag{2}\]  
where \(j\) and \(k\) are the indices in the horizontal and vertical directions, respectively.  
To estimate the parameters \(f_{n,m}\) of the equalizer, we minimize the mean square error (MSE) between output signal of the equalizer \(z[j,k]\) and input signal of the channel \(s[j,k]\). That is, this is a zero-forcing equalizer. However, because the zero-forcing equalizer is designed for 1D interference, its performance in an HDS channel is very low. In other words, signal \(z[j,k]\) cannot be approximated to signal \(s[j,k]\) in 2D ISI. Therefore, instead of approaching signal \(s[j,k]\), signal \(z[j,k]\) approaches desired signal \(d[j,k]\) in (3) by optimizing the MSE in (5) to improve the accuracy of the equalizer.  
\[d[j,k]=\sum_{n=-1}^{1}\sum_{m=-1}^{1}t_{n,m}s[j-n,k-m] \tag{3}\]  
\[\mathbf{T}=\begin{bmatrix}t_{-1,-1}&t_{-1,0}&t_{-1,1}\\ t_{0,-1}&t_{0,0}&t_{0,1}\\ t_{1,-1}&t_{1,0}&t_{1,1}\end{bmatrix} \tag{4}\]  
\[MSE=E\Big{\{}(d[j,k]-z[j,k])^{2}\Big{\}} \tag{5}\]  
If we optimize (5), the parameters for equalizer \(\mathbf{F}\) and target \(\mathbf{T}\) are equal to \(\mathbf{0}\). Therefore, a condition is suggested to avoid \(\mathbf{F}\) and \(\mathbf{T}\) from being equal to \(\mathbf{0}\). The condition is as follows:  
\[\mathbf{c}^{T}\mathbf{t}=\alpha \tag{6}\]  
Figure 2: Equalizer and the target in an HDS system [18].  
where \(\mathbf{t}\) represents a vector form of target \(\mathbf{T}\),  
\[\mathbf{c}^{T}=\begin{bmatrix}0&0&0&0&1&0&0&0&0\end{bmatrix}\text{, and} \tag{7}\]  
\[\alpha=0.85 \tag{8}\]  
In the HDS channel, based on [19], \(\alpha=0.85\) can be shown to achieve the best performance for the HDS channel. Applying the minimum mean square error (MMSE) algorithm, as specified in [18], the parameters for \(\mathbf{F}\) and \(\mathbf{T}\) can be estimated as follows:  
\[\mathbf{t}=\left(\mathbf{S}-\mathbf{D}^{T}\mathbf{R}^{-1}\mathbf{D}\right)^{ -1}\mathbf{c}\left(\mathbf{c}^{T}\left(\mathbf{S}-\mathbf{D}^{T}\mathbf{R}^{- 1}\mathbf{D}\right)^{-1}\mathbf{c}\right)^{-1}\alpha\text{, and} \tag{9}\]  
\[\mathbf{f}=\mathbf{R}^{-1}\mathbf{D}\mathbf{t} \tag{10}\]  
Here, \(\mathbf{S}=E[\mathbf{s}\mathbf{s}^{T}]\), \(\mathbf{R}=E[\mathbf{r}\mathbf{r}^{T}]\), \(\mathbf{D}=E[\mathbf{s}\mathbf{s}^{T}]\), and \(\mathbf{f}\) represents the vector form of equalizer \(\mathbf{F}\). \(\mathbf{s}\) and \(\mathbf{r}\) represent the vector forms of signals \(s[j,k]\) and \(r[j,k]\), respectively.  
In the HDS channel, a point spread function (PSF) is used as the channel impulse response. Because the channel parameters far from the center are close to zero when the size is greater than a \(5\times 5\) array of pixels, we selected a pixel array size of \(5\times 5\) for the equalizer. To design a simple detection method, we selected a target size of a \(3\times 3\) array of pixels. This helps the trellis in horizontal and vertical detections reduce the states to only 36 and 4, respectively.