<figure><img src="image_1.png" /></figure>

HAL
open science

# Low Complexity Convolutional Neural Networks for Equalization in Optical Fiber Transmission

Mohannad Abu-Romoh, Nelson Costa, Antonio Napoli, João Pedro, Yves

Jaouën, Mansoor Yousefi

► To cite this version:

Mohannad Abu-Romoh, Nelson Costa, Antonio Napoli, João Pedro, Yves Jaouën, et al. Low Complexity Convolutional Neural Networks for Equalization in Optical Fiber Transmission. OSA Advanced Photonics Congress 2021, Signal Processing in Photonic Communications (SPPCom) 2021; From the session: Neural Networks Applications for Photonic Systems (SpM5C), Jul 2021, Washington, DC, United States. [hal-04106814](https://hal.inria.fr/hal-04106814)

HAL Id: hal-04106814

https://telecom-paris.hal.science/hal-04106814

Submitted on 25 May 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# Low Complexity Convolutional Neural Networks for Equalization in Optical Fiber Transmission

Mohannad Abu-romoh$^{(1,* )}$, Nelson Costa$^{(2)}$, Antonio Napoli$^{(3)}$, João Pedro$^{(2)}$, Yves Jaouën$^{(1)}$ and Mansoor Yousefi$^{(1)}$

$^{(1)}$ Telecom Paris, 19 Place Marguerite Perey, 91120 Palaiseau, France

$^{(2)}$Infinera Unipessoal Lda, Carnaxide, Portugal $^{(3)}$Infinera, London, United Kingdom

(*) mohannad.aburomoh@telecom-paris.fr

**Abstract:** A convolutional neural network is proposed to mitigate fiber transmission ef-
fects, achieving a five-fold reduction in trainable parameters compared to alternative equal-
izers, and 3.5 dB improvement in MSE compared to DBP with comparable complexity.

© 2021 The Author(s)

1. Introduction

Distortions caused by the Kerr nonlinearity limit the achievable information rates (AIRs) at high powers in optical fiber communication. Mitigation of these distortions is possible using, e.g., digital back-propagation (DBP) [1]. DBP, however, requires knowledge of fiber parameters and topology, and can be computationally expensive in part due to potentially large number of spatial segments. Artificial neural networks (ANNs) offer an alternative approach that might be less complex [2]. Learned digital backpropagation (LDBP) is proposed in [3], in which the model is based on the split-step Fourier method (SSFM), and optimized using the standard learning algorithms for ANNs. In this paper, we consider a convolutional neural network (CNN) equalizer. We note that the SSFM coefficients are repeated in each span. Thus, instead of joint training of all neural network layers, we train a few unique layers that are shared in network depth. With this parameter sharing method, we substantially reduce the number of model trainable parameters.

2. System Model

<figure><img src="image_2.png" /><figcaption>Fig. 1: Block diagram of the end-to-end system model</figcaption></figure>

We assume the polarization division multiplexed (PDM) system shown in Fig. 1. The signal carried by each polar-
ization $U_{x,y}(t,z)$ is generated by quadrature amplitude modulation (QAM) $U_{x,y}(t,0) = \sum_{k=1}^{N_s} s_{x,y}^k p(t-kT_s)$, where
$p(t)$ is the pulse shape as a function of time $t$, $\{s_{x,y}^k\}_k$ are transmitted symbols drawn from a constellation $\mathcal{S}$, $N_s$
is the number of symbols, and $T_s$ is the symbol period. The transmitted signal is propagated through an optical
fiber modeled by the vector nonlinear Schrödinger's system $\frac{\partial U_i}{\partial z} = [-\frac{\alpha}{2} - j\frac{\beta_2}{2} \frac{\partial^2}{\partial t^2} + j\gamma (|U_i|^2 + \frac{2}{3}|U_{\bar{i}}|^2)] U_i$, where
$(i, \bar{i}) = (x, y)$ or $(y, x)$, $z$ is distance, and $\alpha, \beta_2$ and $\gamma$ are, respectively, loss, dispersion and nonlinearity coefficients.

3. Low Complexity Convolutional Neural Network Equalizer

In CNNs, each layer maps the input vector $\mathbf{X}^{(l-1)}$ to an output vector $\mathbf{X}^{(l)} = \Phi(\tilde{\mathbf{W}}^{(l)} * \mathbf{X}^{(l-1)})$ by convolution multiplication with a kernel $\mathbf{W}^{(l)}$. The main idea of this paper is to exploit the similarity between the neural network function and the un-folding of the SSFM, since both involve alternating between linear and nonlinear operations in their functions. It is hence possible to initialize the deep CNN parameters to perform SSFM; This approach has been studied and is referred to as model-based neural networks. We extend the model-based design of neural networks by assuming that some of the training parameters can be shared between multiple layers. Fig.2 shows the proposed CNN equalizer with parameter sharing and the corresponding SSFM with $M = 2$ steps per span. Since the linear and nonlinear operations of SSFM (i.e. $A_m$ and $\sigma$ in Fig.2, respectively) are repeated in each span in the optical fiber, we are able to identify 4 layers with unique parameters, namely $W_1 \hat{=} G_1 A_1$, $W_2 \hat{=} A_2 G_2 A_3$, $W_3 \hat{=} A_4 G_1 A_1$ and $W_4 \hat{=} A_4$. Overall, the number of uniquely defined layers is $M + 2$ regardless of the number of
<figure><img src="image_3.png" /><figcaption>Fig. 2: An illustration of the proposed neural network architecture. The top branch corresponds to asymmetric DBP, and the bottom branch to the proposed CNN. $U_{x,y}^{(0)}$ and $U_{x,y}^{(L)}$ refer to the signals at the transmitting and receiving ends of the fiber, where L is the number of segments.</figcaption></figure>

spans in the optical link. The nonlinear activation function used in our CNN is $\Phi(\mathbf{X}) = \mathbf{X}e^{j(||\mathbf{X}||^2 + \frac{2}{3}||\mathbf{Y}||^2)}$ and $\Phi(\mathbf{Y}) = \mathbf{Y}e^{j(||\mathbf{Y}||^2 + \frac{2}{3}||\mathbf{X}||^2)}$ for each polarization. In our model, amplification is applied after each segment such that $\prod_{k=1}^{M} G_k = 1$. With this, we provide the neural network the ability to adjust the intensity of the signal, and hence, the nonlinear phase rotation without the need of trainable parameters in the activation function.

## 4. Results

In our simulations, we generate dual-polarized 16-QAM signals at 32 Gbaud using root raised cosine pulse-shaping (roll-off factor = 0.1). We set the fiber length $L_f=1000$ km, and the number of spans $N_{sp}=10$. Forward-propagation is simulated with 50 steps/SPAN, and data is initially sampled at a high sampling rate of 16 samples/symbol. On the receiver side, the signal is down-sampled to 2 samples/symbol before being processed by either CNN or DBP algorithms. We use Keras API built on TensorFlow to implement and optimize the neural network parameters. During the training of the model, examples consisting of input-output pairs were randomly chosen from the set of launch powers $P \in \{0,1,2,3,4\}$ dBm. For each point in Fig. 3, we generate 3300 input-output pairs for testing. We express the performance gain in terms of the normalized mean squared error (MSE) defined as $\frac{||\mathbf{X}-\hat{\mathbf{X}}||^2+||\mathbf{Y}-\hat{\mathbf{Y}}||^2}{||\mathbf{X}||^2+||\mathbf{Y}||^2}$. The proposed CNN with 1 layer/SPAN achieved MSE of 25 dB at 2 dBm launch power which corresponds

to 3.5 dB improvement compared to the best performance attained by DBP with optimized step-size and equivalent complexity. For 2 layers/SPAN, the peak MSE value for the CNN is 27 dB measured at 5 dBm launch power, which is 3 dB higher than DBP with similar complexity. Comparing to LDBP (asterisks in Fig. 3), a small gain in MSE is observed. We explain this gain by the neural networks ability to generalize better with fewer training parameters. In terms of training complexity, our model uses 3 uniquely defined layers comparing to 11 layers for LDBP, in the 1 layer/SPAN setup, and 4 uniquely defined layers comparing to 21 layers for LDBP, in the 2 layers/SPAN setup.

<figure><img src="image_4.png" /><figcaption>Fig. 3: Comparison between compensation algorithms.</figcaption></figure>

## 5. Conclusion

A parameter sharing method is proposed to reduce the training complexity of CNNs for equalization in optical fiber. The proposed approach yields 3 – 3.5 dB gain in MSE compared to optimized DBP with comparable complexity, and five-fold reduction in number of trainable parameters compared to LDBP at the same MSE.

## Acknowledgement

This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 813144.

## References

1. A. Napoli, Z. Maalej, V. A. Sleiffer, M. Kuschnerov, D. Rafique, E. Timmers, B. Spinnler, T. Rahman, L. D. Coelho, and N. Hanik, "Reduced complexity digital back-propagation methods for optical communication systems," *Journal of lightwave technology*, vol. 32, no. 7, pp. 1351-1362, 2014.

2. S. Zhang, F. Yaman, E. Mateo, and Y. Inada, "Neuron-network-based nonlinearity compensation algorithm," in *2018 European Conference on Optical Communication (ECOC)*, 2018, pp. 1-3.

3. R. M. Bütler, C. Häger, H. D. Pfister, G. Liga, and A. Alvarado, "Model-based machine learning for joint digital backpropagation and pmd compensation," *Journal of Lightwave Technology*, vol. 39, no. 4, pp. 949-959, 2021.