# Equalization Methods for Out-of-Band Nonlinearity Mitigation in Fiber-Optic Communications
## 4 Equalization Algorithms
### Standard Equalization Algorithms  
There are several important aspects that require consideration when choosing an equalization algorithm. The method should be able to track the fast variations of the ISI coefficient, be robust to symbol estimation errors, and be computationally efficient. We will now discuss a few of these algorithms. The figures of merit to describe the quality of channel estimation is the improvement in post-equalization SNR, and the computational cost of the algorithm. For the reader's convenience, we also include a short description of the algorithms themselves. For the clarity of presentation, the algorithms are described for the scalar case, keeping in mind that the extension to polarization-multiplexed transmission is usually straightforward. In the following, \(\hat{s}_{n}\) denotes the equalized signals, \(\tilde{H}_{n}\) is a vector containing the equalizers' taps, and \(D(s_{n})\) represents the hard decision regrading the value of the \(n\)-th data symbol (i.e., in the absence of errors \(D(s_{n})=a_{n}\)). 1. _Least mean squares (LMS) algorithms_ The LMS algorithm is essentially a gradient descent method, where the evaluation of the channel filter improves over time, until it converges. The LMS algorithm has been used in [6; 7] to mitigate  
Figure 3: A block-diagram description of a general equalizer. All equalizers discussed in this section follow the same structure, with the differences being in the operation of the channel estimation and equalization blocks. PPRN. The benefit of using dual-polarization decision to improve its performance was shown in [20; 21; 35]. The algorithm can be found in textbooks [36], and is summarized as \[\hat{s}_{n} :=\bar{H}_{n}^{+}\hat{s}_{n}\] (8) \[\epsilon_{n} :=D(s_{n})-\hat{s}_{n}\] (9) \[\bar{H}_{n+1} :=\bar{H}_{n}+\mu\frac{\epsilon_{n}^{*}s_{n}}{\hat{s}_{n}^{*}s_{n}}.\] (10) Here \(\hat{s}_{n}\) is a vector containing the received signal samples \(s_{n}\) at times \(n\ldots n+N-1\), where \(N\) is the equalizer's number of taps. The parameter \(\mu\) controls the speed of convergence, and it is chosen empirically so as to produce the best equalization results. The main advantage of this algorithm is its computational efficiency. However, its tracking speed is fairly slow. In practice, the LMS algorithm has been shown to provide reasonable mitigation of PPRN, but not of higher order ISI contributions. 2. _Recursive Least squares (RLS) algorithms_ The RLS algorithm is usually characterized by a higher convergence speed than LMS, which generally implies improved tracking speed of fast channel variations. It has been used to mitigate PPRN [6; 8] as well as higher order NLIN induced ISI contributions, although the latter was only demonstrated in idealized (error-free) scenarios [17]. The algorithm can be summarized as \[\hat{r}_{n} :=\mathbf{P}_{n}\hat{s}_{n}\] (11) \[\mathbf{P}_{n+1} :=\frac{1}{\lambda}\left(\mathbf{P}_{n}-\frac{\hat{r}_{n}\hat{r}_ {n}^{\dagger}}{\lambda+\hat{r}_{n}^{\dagger}\hat{r}_{n}}\right)\] (12) \[\bar{H}_{n+1} :=\bar{H}_{n}+\frac{\hat{r}_{n}\epsilon_{n}^{*}}{\lambda+\hat{r}_ {n}^{\dagger}\hat{r}_{n}},\] (13) where the error metric \(\epsilon_{n}\), the signal vector \(\hat{s}_{n}\), and the equalized signal \(\hat{s}_{n}\) are defined as in Equations (8) and (9). The parameter \(\lambda\) is called the algorithms 'forget factor', and it determines the algorithm's tracking accuracy. In general, RLS is considered to be more computationally expensive than LMS, because it involves matrix multiplications. In the case of NLIN mitigation, the equalizer's number of taps is fairly small, corresponding to small matrices. In practice, we found that in relevant scenarios, the computational costs of RLS and LMS are similar (see Figure 4b). 3. _Window-averaging algorithms_ Window-averaging algorithms use the fact that the NLIN is temporally correlated whereas other noise sources are not.