# Deep-Learning for generic blind-Joint Channel Equalization and Power Amplifier Post-distortion

Chouaib Farhati, Souhaila Feki, Hmaied Shaiek, Ali Dziri, Abdeldjalil Aïssa

El Bey, Fatma Abdelkefi

► To cite this version:

Chouaib Farhati, Souhaila Feki, Hmaied Shaiek, Ali Dziri, Abdeldjalil Aïssa El Bey, et al. Deep-Learning for generic blind-Joint Channel Equalization and Power Amplifier Post-distortion. IEEE Access, 2023, 11, pp.104754-104762. 10.1109/ACCESS.2023.3317364. hal-04213738

HAL Id: hal-04213738

https://imt.hal.science/hal-04213738v1

Submitted on 11 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Received 4 September 2023, accepted 15 September 2023, date of publication 20 September 2023, date of current version 28 September 2023.

Digital Object Identifier 10.1109/ACCESS.2023.3317364

RESEARCH ARTICLE

# Deep-Learning for Generic Blind-Joint Channel Equalization and Power Amplifier Post-Distortion

CHOUAIB FARHATI<sup>1</sup>, SOUHAILA FEKI<sup>1</sup>, HMAIED SHAIEK<sup>1,2</sup>, (Member, IEEE), ALI DZIRI<sup>1,2</sup>, (Member, IEEE), ABDELDJALIL AÏSSA EL BEY<sup>1,3</sup>, (Senior Member, IEEE), AND FATMA ABDELKEFI<sup>1</sup>, (Senior Member, IEEE)

Corresponding author: Chouaib Farhati (chouaib.farhati@supcom.tn)

**ABSTRACT** One of the major problems faced in digital communication systems is Inter-Symbol Interference (ISI), induced by the propagation channel in the single carrier based systems. Classic digital equalization techniques based on pilot training sequences become tedious in the presence of nonlinear power amplifiers. The existing techniques for digital pre-distortion need a high transmitter computation complexity and those for the post-distortion require pilot overhead. In this review, we focus on fully generic blind processing for both channel equalization and power amplifier post-distortion. We propose a receiver based on two complex-valued neural networks (CV-NN). The first CV-NN is dedicated to generic blind equalization (GBE) to mitigate ISI. The second one is used for generic blind post-distortion compensation (GBPDC) of the power amplifier nonlinearity (PANL). The GBE and the GBPDC have no prior information about the transmission channel, the used constellation, and the PA model. For the first CV-NN, we consider an updated probability density fitting (PDF) based criteria, corresponding to many assumed possible constellations, that are used jointly with an automatic modulation classification (AMC) based on the k-nearest neighbors (KNN) algorithm. For the second CV-NN, we use the final updated PDF criterion resulting from the first CV-NN training process. Numerical results show that our generic blind Deep learning-based signal receiver formed with the two CV-NNs is effective in alleviating the two coupled signal distortions: the ISI and the PANL. Compared with the state-of-the-art methods, banking on a supervised post-distortion compensation and channel equalization, the proposed generic blind DL-based scheme exhibits good detection performance.

**INDEX TERMS** Automatic modulation classification, blind channel equalization, neural network, post-distortion, power amplifier, pre-distortion, probability density fitting.

## I. INTRODUCTION

The nonlinearity of power amplifiers (PA) remains one of the serious constraints for the fifth generation (5G) and beyond cellular systems since signals have extremely high peak-to-average power ratio (PAPR). Moreover, channel estimation and equalization remain hot topics in 5G and beyond use cases and applications in various hard propagation conditions. To avoid the overhead data in the pilot-aided schemes for channel estimation and equalization, the blind

The associate editor coordinating the review of this manuscript and approving it for publication was Walid Al-Hussaibi.

approach has been widely investigated to reduce the ISI due to multipath propagation.

Moreover, the PANL can be mitigated in two ways: linearization techniques at the transmitter side and post-distortion approach at the receiver side. In the literature, various linearization methods are proposed such as feed-forward linearization [1], feedback linearization [2] and the digital pre-distortion approach. The basic concept of the pre-distortion approach is to add a distorter at the transmitter side, based on an approximation of the PA inverse function. The main challenge provided by the digital pre-distortion approach is its high computational complexity that requires

<footnote n="1">¹MEDIATRON Laboratory, High School of Communications of Tunis, University of Carthage, Ariana 2083, Tunisia</footnote>

<footnote n="2">²National Conservatory of Arts and Crafts, 75003 Paris, France</footnote>

<footnote n="3">³IMT Atlantique, Lab-STICC, UMR CNRS 6285, F-29238 Brest, France</footnote>
more energy and resources, leading to its limited use on the base stations side. To overcome the pre-distortion challenges, post-distortion has been widely investigated. It manages the PANL compensation on the receiver side. This approach is proposed for millimeter wave communication systems in 5G and beyond [3]. In [3], two neural networks (NN) are integrated, the first for ISI mitigation and the second for PANL avoidance purposes. Authors in [4] propose a deep learning-based receiver with PA. In [5] a Volterra series-based convolution neural network (CNN) is suggested to mitigate PANL. In the literature, several studies based on post-compensation have been carried out [6], [7]. In all these digital post-distortion-based solutions discussed above, the two coupled distortions: linear ISI and PANL, are addressed separately with non-blind training. In addition, we assume a single carrier transmission system where the spectrum emission constraints are satisfied.

In wireless communication, a multi-path channel introduces ISI. Blind equalization is one of the techniques applied to mitigate ISI, it avoids bandwidth waste resulting from training data. In the literature, various blind equalizers based on high-order statistical (HOS) properties are proposed as the constant modulus algorithm (CMA) [8] and the multi-modulus algorithm (MMA) [9]. Other blind equalizers based on probability density fitting are also developed as the stochastic quadratic distance (SQD) [10] and the multi-modulus stochastic quadratic distance (MSQD-$\ell_p$) [11]. PDF-based equalizers that exploit the full range of signal distribution surpass equalizers based on HOS properties that exploit a part of signal features [11]. For this reason, we are interested in PDF-based criteria in this paper.

In the last decade, the neural network has been widely operated in signal processing such as channel estimation and equalization. In the literature, several works were investigated to build blind neural equalizers like the neural network constant modulus algorithm (NNCMA) [12] and the neural network multi-modulus algorithm (NNMMA) [13]. These algorithms outperform their linear versions: CMA and MMA. In the same way, we have previously implemented the SQD and MSQD-$\ell_p$ PDF based criteria with a neural network [14] and have shown that using a neural network for these criteria outperforms their linear versions.

Numerous neural network architectures were considered for blind equalization. We can mention, the feedforward equalizer (FFE) [12], the feedforward with decision feedback equalizer (FFE-DF) [15], the recurrent neural network equalizer (RNNE) [16] and variational autoencoders [17]. All these previous architectures process the real and imaginary parts of the signal separately, thus ignoring the correlation between the real and imaginary parts. In addition, the CV-NN [12] processes the real and imaginary parts together, preserving the correlation between them and consequently leading to a robust convergence.

In some communication scenarios, the receiver has no information on the transmission channel and the used constellation. Thus a GBE is used to cancel ISI. After equalization,

and before taking the final decision and exploiting the received data, the receiver classifies the GBE output by an AMC algorithm to reach the transmitted constellation.

In the state of the art, some works consider jointly GBE and AMC. We can cite [18] where a multi-branch architecture is used, each branch is linked to a specific constellation. The branch provides the smallest error is considered. Other approaches use GBE based on CMA and MSQD-lp algorithms [19], [20]. They use this criterion by reading with the quadratic phase shift keying (QPSK) constellation to equalize any constellation belonging to the phase shift keying modulation (PSK) or the quadrature amplitude modulation (QAM). The resulting constellation is scaled to ensure the targeted transmit power. All the cited works proposed a linear GBE followed by an AMC process.

In [21], we have implemented a joint neural GBE using CV-NN, and AMC based on a KNN classifier with the fourth-order cumulants which deal with quadrature amplitude modulation (QAM) [22], [23].

Moreover, since a PA has been included in a major part of the communication systems, in this paper we focus on its effect. In the literature, there are many PA models such as the polynomial model [24], [25], Rapp's model [26], [27] and Saleh's model [28]. The nonlinearity behavior of these PA models is usually characterized by their amplitude modulation-amplitude modulation (AM-AM) and amplitude modulation-phase modulation (AM-PM) conversions. In this paper, we consider the modified Rapp model [29], which has both AM-AM and AM-PM conversion characteristics.

Employed orthogonal frequency division multiplexing (OFDM) is used in communication systems such as 5G and beyond [3]. However, there are other systems that still use the single carrier communication system such as maritime communication, satellite communication, and the Internet of Things (IoT) technologies. The work proposed here considers single input single output (SISO) communications and we are currently working on its extension to MIMO systems.

In this work, we propose a fully blind Deep Learning (DL)-based signal receiver with two sequential CV-NNs banking on the probability density fitting (PDF) criterion. The first one mitigates linear ISI and the second one compensates PANL. We also propose a generic blind DL-based signal receiver that mitigates the ISI by the neural network GBE (NNGBE) and the PANL by the neural network GBPDC (NNGBPDC).

Accordingly, the main contributions of this paper are resumed as follows:

* Joint simultaneously blind channel equalization, and blind post-distortion compensation.

* Joint simultaneously generic blind channel equalization, and generic blind post-distortion compensation.

To our knowledge, this is the first time that the PDC is processed blindly.

The rest of this paper is organized as follows. Section II describes the system and signal models. Section III introduces the blind DL-based signal receiver. The generic blind DL-based signal receiver is described in section IV.
Simulation results in a complete transmitter/receiver chain are discussed in section V. Finally, Section VI concludes this work.

## II. SYSTEM MODEL

### A. SIGNAL MODEL

In the rest of this paper, we consider the following parameters and notations. We note that lowercase letters, e.g. $x$, and bold lowercase letters, e.g. $\mathbf{x}$, denote scalars and vectors respectively. $\Re(\cdot)$ or $(\cdot)_r$ denotes the real part of a complex number, and $\Im(\cdot)$ or $(\cdot)_i$ denotes the imaginary part of a complex number. * denotes the conjugate operator.

Our base-band communication system is detailed in Fig.1.

<figure><img src="image_2.png" /><figcaption>FIGURE 1. Transmission system base-band model with a power amplifier, and DL-based signal receiver.</figcaption></figure>

* $s(n)_{n \in \mathbb{Z}}$ is a normalized M-QAM sequence of independent and identically distributed (i.i.d) complex symbols,

* $x(n)$ is the PA input and $\alpha$ is a multiplicative factor to adjust the average power of the signal at the PA input,

* $a(n)$ is the PA output,

* $\mathbf{h} = [h_0, h_1, \dots, h_{L_h-1}]$ is a finite impulse response (FIR) of the transmission channel,

* $b(n)$ is an additive white Gaussian noise (AWGN),

* $z(n)$ is the channel output,

* $\mathbf{i}_1(n)$ and $o_1(n)$ are the input and the output of the first CV-NN equalizer at a given time $n$, respectively,

$$o_1(n) = G_1(\mathbf{i}_1(n), \mathbf{w}_1, \mathbf{w}_2), \quad (1)$$

where $G_1$ is the first CV-NN three layers global function, $\mathbf{i}_1(n) = [y(n), y(n-1), \dots, y(n-L_1+1)]^T$ is a vector of $L_1$ complex data samples applied at its input layer. $w_1$ is the matrix of synaptic weights between the input and hidden layers and $w_2$ is the matrix of synaptic weights between the hidden and output layers.

* $\mathbf{i}_2(n)$ and $o_2(n)$ are the input and the output of the second CV-NN at a given time $n$, respectively:

$$o_2(n) = G_2(\mathbf{i}_2(n), \mathbf{u}_1, \mathbf{u}_2), \quad (2)$$

where $G_2$ is the second CV-NN three layers global function, $\mathbf{i}_2(n) = [p_1(n), \dots, p_{L_2+1}(n)]$ is a vector of $L_2$ complex data samples applied at its input layer, with $p_k(n) = o_1(n)|o_1(n)|^k$. $\mathbf{u}_1$ is the matrix of synaptic weights between the input and hidden layers and $\mathbf{u}_2$ is the matrix of synaptic weights between the hidden and output layers.

### B. CV-NN MODEL

The first CV-NN structure is shown in Fig.2. In the sequel, we assume $N_k$ neurons in the $k^{th}$ layer and we denote by $\phi_j^k$ and $\nu_j^{k+1}$ is the input and the output of the $j^{th}$ neuron, such that:

$$\phi_j^k = \sum_{i=1}^{N_k-1} w_{ij}^k \nu_i^k + \theta_j^k \quad \text{and} \quad (3)$$

$$\nu_j^{k+1} = f^k(\text{Re}(\phi_j^k)) + jf^k(\text{Im}(\phi_j^k)), \quad (4)$$

<figure><img src="image_3.png" /><figcaption>FIGURE 2. CV-NN architecture.</figcaption></figure>

where $\nu_i^k$ is the $k^{th}$ layer input, $w_{ij}^k$ is the weight between the $i^{th}$ neuron in the $k^{th}$ layer and the $j^{th}$ neuron in the $(k+1)^{th}$ layer, $\theta_j^k$ and $f^k(\cdot)$ are the $k^{th}$ layer bias and activation function, respectively. The CV-NN is trained using the complex backpropagation (CBP) algorithm [30].

The second CV-NN has the same structure as the first CV-NN with the appropriate inputs defined in the previous subsection.

### C. POWER AMPLIFIER MODEL

In this work, we assume the memoryless nonlinearity introduced by the modified Rapp's PA model. This PA is commonly used to model solid-state power amplifiers (SSPAs) [31] and exhibits both the AM-AM and AM-PM conversion characteristics. The signal at the output of the PA model can be written as follows:

$$a(n) = F_a(\rho(n)) \exp(j(\varphi(n) + P_a(\rho(n))), \quad (5)$$

where $\rho(n)$ and $\varphi(n)$ are respectively the input signal ($x(n)$) modulus and phase. $F_a(\cdot)$ is the AM/AM conversion function,
and $P_a(\cdot)$ is the AM/PM conversion function, expressed, for the modified Rapp model, as follows:

$$F_a(\rho(n)) = \frac{g\rho(n)}{\left(1 + \left|\frac{g\rho(n)}{V_{sat}}\right|^{2p}\right)^{\frac{1}{2p}}}, \quad P_a(\rho(n)) = \frac{k\rho(n)^q}{\left(1 + \left|\frac{\rho(n)}{u}\right|^q\right)^q}, \qquad (6)$$

where g is the linear gain, $V_{sat}$ is the saturation voltage and p is a smoothness factor that controls the transition from the linear region to the saturation one. k, u and q respectively represent the adjustment parameters. In this paper, we assume that $V_{sat} = 1.9$, g = 16, k = -345, u = 0.17, p = 1.1 and q = 4 [26], [32]. This PA is applied for modeling sub-6GHz communications for 4G and 5G systems [3]. In this paper, we ignore the memory effects because the PA response is assumed to be constant over the signal band [33].

The PA nonlinear conversion characteristics can produce distortions in the constellation scheme, as well as spectral regrowth, degrading the system performance [4]. In practice, to reduce the nonlinearity effects, the PA is operated at a given input back-off (IBO) level from its saturation point [3]. In the logarithmic scale, the IBO is defined by $IBO = 10 \log_{10} \left(\frac{V_{sat}}{P_x}\right)$, where $P_x$ is the average power of the PA input signal $x(n) = \alpha s(n)$. We remind that $\alpha$ is a scaling factor adjusted to reach the targeted IBO value.

### III. BLIND DL-BASED SIGNAL RECEIVER

Communications systems integrate usually power amplifiers, whose non-linear characteristics may affect transmission quality. The received signal is impacted by two coupled effects: ISI and PANL. ISI is often mitigated by the receiver, but the PANL is compensated in two ways: either by pre-distortion at the transmitter or with post-distortion at the receiver side. In this section, we will detail the blind DL-based signal receiver that we propose. It has the advantage to equalize the wireless transmission channel and compensate the PANL.

The receiver includes two sequential CV-NNs as shown in Fig.3. The first one aims to mitigate ISI and the second one aims to compensate PANL.

<figure><img src="image_4.png" /><figcaption>FIGURE 3. Blind DL-based receiver architecture.</figcaption></figure>

The Two sequential CV-NNs are trained by employing the MSQD-$\ell_p$ algorithm [11]. Specially for $p = 2$, the MSQD-$\ell_2$

has the following cost function:

$$J_{MSQD-\ell_2}(\mathbf{w}) = -\frac{1}{N_s} \sum_{k=1}^{N_s} K_{\sigma_o} (|o_r|^2 - |s_r(k)|^2) \\ - \frac{1}{N_s} \sum_{k=1}^{N_s} K_{\sigma_o} (|o_i|^2 - |s_i(k)|^2), \qquad (7)$$

where $N_s$ is the number of complex symbols in the considered constellation and $K_{\sigma_o}$ is a Gaussian kernel with zero mean and variance $\sigma_o$ which is referred to kernel width. o and s are the equalized output and the constellation symbols, respectively.

The blind equalization criterion based on PDF tries to estimate the distribution of the transmitted data at the receiver side. It minimizes the distance between the signal PDF at the equalizer output and the emitted constellation during iterations.

#### A. ISI CANCELLATION (CV-NN 1)

In the first stage, CV-NN 1 is trained according to (7) in order to mitigate ISI. CBP was investigated for the training process and we consider the stochastic gradient descent (SGD) algorithm to update the network weights, such that:

$$w_{ij}^{k}(n+1) = w_{ij}^{k}(n) - \mu \frac{\partial J_{MSQD-\ell_2}}{\partial w_{ij}^{k}(n)}. \qquad (8)$$

The gradient is calculated as in [14]. For simplification purposes, we introduce two auxiliary parameters $Q_r$ and $Q_i$ which are defined as follows:

$$Q_r = \frac{1}{N_s} \sum_{k=1}^{N_s} \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(|o_r(n)|^2 - |s_r(k)|^2)^2}{2\sigma^2}} \\ \times \frac{(|o_r(n)|^2 - |s_r(k)|^2)}{\sigma^2},$$

and

$$Q_i = \frac{1}{N_s} \sum_{k=1}^{N_s} \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(|o_i(n)|^2 - |s_i(k)|^2)^2}{2\sigma^2}} \\ \times \frac{(|o_i(n)|^2 - |s_i(k)|^2)}{\sigma^2}.$$

Let $\delta_p^o = Q_r o_r f^o(\varphi_r^o) + j Q_i o_i f^o(\varphi_i^o)$.

Further, the weights of the output layer are updated as the following:

$$w_j^o(n+1) = w_j^o(n) - \mu \delta_p^o I_{pj}^*, \qquad (9)$$

where $\varphi^o$ is the input of the output layer. $I_{pj}$ is the output of the hidden layer. $f^o(\cdot)$ is the activation function of the output layer. $f^o(\cdot)$ is the derivative of the activation functions of the output layer.

The weights of the hidden layer are updated as follows:

$$w_{ji}^{h}(n+1) = w_{ji}^{h}(n) - \mu \delta_{pj}^{h} y_{pi}^{*}, \qquad (10)$$

$$\delta_{pj}^{h} = f_{j}^{h'}(\varphi_{r}^{h}) \mathrm{Re}(\delta_{p}^{o} w_{j}^{o*}) + j f_{j}^{h'}(\varphi_{i}^{h}) \mathrm{Im}(\delta_{p}^{o} w_{j}^{o*}), \qquad (11)$$
where $\phi^h$ is the input of the hidden layer and $y_{pi}$ is the input of the network. $f_j^h(.)$ is the hidden layer derivative activation function for the $j^{th}$ neuron.

## B. PANL COMPENSATION (CV-NN 2)

After training CV-NN1, we obtain its optimal parameters. Then, the training of CV-NN 2 is performed, and where the training process is the same as the CV-NN1 (CBP, SGD) one. The input of CV-NN2 is a polynomial constructed from the output of CV-NN1 as described in equation (2).

Once the learning is done for both networks and the optimal parameters are procured, both models are performed online to cancel ISI in the first stage and compensate PANL in the second stage.

## IV. GENERIC BLIND DL-BASED SIGNAL RECEIVER

We now assume that the receiver has no information about the used constellation. So, it becomes a generic receiver for different possible M-QAM constellations and it is built from two sequential CVNNs with an AMC block as shown in Fig.4. The first CV-NN mitigates ISI, and the second one compensates PANL.

<figure><img src="image_5.png" /><figcaption>FIGURE 4. Generic blind DL-based receiver architecture.</figcaption></figure>

We have trained the first CV-NN with a new multi-criteria generic cost function as a sum of several variations of equation (7), one for each constellation order multiplied by an updated penalty factor. So, the criterion will be:

$$J_{MC-MSQD-\ell_{2gen}}(\mathbf{w}) = \sum_{m=1}^{C} \alpha_m J_{MSQD-\ell_{2m}}(\mathbf{w}), \quad (12)$$

where C is the number of the constellation orders.

The penalty factor $\alpha_m$ in (12) is updated in each iteration according to equation (13) such that we reach, over the iterations, the transmitted constellation cost function:

$$\alpha_m = \text{sigmoid}(\text{class}_m), \quad (13)$$

where class_m is the number of times where the $m^{th}$ constellation is successful in the classification result after the classification step. At the end of the CV-NN 1 training process, (13) will converge to a specific cost function that deals with the classified constellation. This cost function will be used for the CV-NN 2 training.

## A. ISI CANCELLATION (CV-NN 1)

In the first step, CV-NN 1 is trained according to (12) in order to mitigate ISI. CBP was investigated for the training process and we consider the stochastic gradient descent (SGD) algorithm to update the network weights, such that:

$$w_{ij}^{k}(n+1) = w_{ij}^{k}(n) - \mu \frac{\partial J_{MC-MSQD-\ell_{2gen}}}{\partial w_{ij}^{k}(n)}, \quad (14)$$

In the following, we will concentrate on the gradient of (12). To do so, we use the gradient of (7), since the gradient of (12) is a sum of the (7) gradient one for each constellation order.

The weights of the output layer are updated as:

$$w_j^o(n+1) = w_j^o(n) - \mu \sum_{m=1}^{C} \alpha_m \delta_{pm}^o I_{pj}^*, \quad (15)$$

where $I_{pj}$ is the output of the hidden layer and $\delta_{pm}^o$ is calculated as the same in subsection III-A for each $m^{th}$ constellation order.

Similarly, the weights of the hidden layer are updated as:

$$w_{ij}^{h}(n+1) = w_{ij}^{h}(n) - \mu \sum_{m=1}^{C} \alpha_m \delta_{pjm}^{h} y_{pi}^*, \quad (16)$$

where $y_{pi}$ is the input vector of the CV-NN and $\delta_{pjm}^h$ is calculated as the same in subsection III-A for each $m^{th}$ constellation order at the $j^{th}$ neuron.

In order to update the cost function, each equalized symbol is classified using the KNN algorithm with fourth-order cumulant [34], [35] as features. So that we execute jointly the equalization and the AMC. The expressions of the $p^{th}$ order cumulant and moment are given by:

$$C_{pq} = \text{cum}(x^{p-q}(x^*)^q) \\ \text{and} \\ M_{pq} = \mathbb{E}[x^{p-q}(x^*)^q],$$

where $\mathbb{E}[\cdot]$ is the expectation operator.

In particular, we will consider:

$$C_{40} = M_{40} - 3M_{20}^2 \\ C_{42} = M_{42} - |M_{20}|^2 - 2M_{21}^2,$$

as they are more suitable for M-QAM modulations [35]. A baseline containing $C_{40}$ and $C_{42}$ values for noisy symbols according to various values of signal-to-noise ratio (SNR) and belonging to {16, 32, 64, 128, 256}-QAM modulations is prepared.

The classifier follows the following steps:

* Calculate $S_1 = C_{40} + C_{42}$ the sum of the last 1000 equalized symbols.

* Calculate the Euclidean distances between $S_1$ and each sum in the reference base.

* A set of the $k$ nearest neighbors from the baseline is created.
* Select the mostly repeated order in the previous, as each neighbor matches a specific constellation order.

Finally, according to the AMC, we update the penalty factors following (13) and update (12).

## B. PANL COMPENSATION (CV-NN 2)

Once CV-NN 1 is trained and converging, we continue training CV-NN 2 with the resulting cost function from CV-NN 1. The output of CV-NN 1 with a polynomial form is the CV-NN 2 input as described in equation (2) and we use the same training process (CBP, SGD).

As in the blind approach, once the learning and classification phases are done for both networks and the optimal parameters are obtained, both models are performed online to mitigate ISI and classify the used constellation in the first step, and compensate PANL in the second step.

## V. SIMULATIONS RESULTS

The blind DL-based signal receiver and the generic blind DL-based signal receiver are conceived with two sequential three layers CVNNs: The first CV-NN neuron number is $L_i = 30$ for the input layer, $L_h = 16$ for the hidden layer, and $L_o = 1$ for the output layer, for the second CV-NN $L_i = 5$, $L_h = 9$, and $L_o = 1$. We assume the following activation function:

$$f(x) = x + \beta * \tanh(x),$$

where $\beta \in [0, 1]$, we choose in the simulations $\beta = 0.1$, and tanh is the trigonometric function. In addition, we tested the performance of the proposed post-processing structures two channels models. The first model is a stationary one corresponding to a typical multi-path radio channel [14]:

$$h_r(z) = (0.0410 + j0.0109) + (0.0495 + j0.0123)z^{-1} + (0.0672 + j0.0170)z^{-2} + (0.0919 + j0.0235)z^{-3} + (0.7920 + j0.1281)z^{-4} + (0.3960 + j0.0871)z^{-5} + (0.2715 + j0.0498)z^{-6} + (0.2291 + j0.0414)z^{-7} + (0.1287 + j0.0154)z^{-8} + (0.1032 + j0.0119)z^{-9}.$$

The second channel model $h_v$, is a time-varying multi-path fading channel simulated using the Rayleigh distribution with the parameters described in Table 1:

<table><caption>TABLE 1. Parameters of time-varying multi-path fading channel.</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>Sampling rate</td><td>3 MHz</td></tr><tr><td>Path delays</td><td>[0, 310, 710, 1090, 1730, 2510] ns</td></tr><tr><td>Average path gains</td><td>[0, -1, -9, -10, -15, -20] dB</td></tr><tr><td>Maximum Doppler shift</td><td>100 Hz</td></tr></tbody></table>

These parameters correspond to a mobile speed of 30 meter per second and a carrier frequency of 2 gigahertz.

We have initialized the two matrices of synaptic weights $w^o[L_o, L_h]$ and $w^h[L_h, L_i]$ with small values except for $w^o[(L_o + 1)/2, (L_h + 1)/2]$ and $w^h[(L_h + 1)/2, (L_i + 1)/2]$ that were set to 1.5.

The total number of symbols used during the training step is 70000. An important factor for convergence during the

training phase is the learning rate $\mu$. It is fixed to $10^{-3}/P_y$, where $P_y$ is the received signal power. All the compared algorithms in this paper have the same convergence speed and in all figures, we draw the BER values after the convergence.

We have taken CV-NN 1 trained in nonblind mode followed by analytic compensation of PANL as a DL-based signal receiver benchmark. To compute the analytical compensation, we use the Newton's approximation algorithm such in [36] to numerically calculate the value of the inverse of the AM/AM conversion. We then use this calculated value to obtain the AM/PM conversion to be extracted from the phase of the signal at the output of CV-NN 1. Thus, we have simulated the DL-based signal receiver composed of CV-NN 1 and CV-NN 2 using 16-QAM and 32-QAM signal constellations in both blind and generic versions. The different BER figures presented in this paper are taken with respect to various signal-to-noise ratio (SNR) values [37], [38].

For nonlinear effects, We have implemented the modified Rapp model described in Section II-C. The polynomial non-linearity at the input of CV-NN 2 $L_2$ is fixed to 5 containing the even and the odd orders. The computational complexity order for both CV-NN 1 and CV-NN 2 is the same. It is a linear function of neuron number for each network layer.

Computation carried for CV-NN 1 and CV-NN 2 are summarized in table 2.

<table><caption>TABLE 2. Computational complexity.</caption><thead><tr><td>Equalizer</td><td>X</td><td>Exponents</td></tr></thead><tbody><tr><td>CV-NN 1</td><td>3L<sub>h</sub>(L<sub>i</sub> + L<sub>o</sub>) + 2L<sub>h</sub></td><td>2N<sub>s</sub></td></tr><tr><td>CV-NN 2</td><td>3L<sub>h</sub>(L<sub>i</sub> + L<sub>o</sub>) + 2L<sub>h</sub></td><td>2N<sub>s</sub></td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>FIGURE 5. BER versus SNR and IBO using $h_r$ in the case of a 16-QAM signal in a blind mode.</figcaption></figure>

## A. BLIND DL-BASED SIGNAL RECEIVER

Fig. 5 confirms that our blind DL-based signal receiver with a 16-QAM received signal and a radio channel, outperforms
the blind ISI cancellation without PANL compensation, especially for higher SNR values. The BER Performance is also affected by the IBO. In fact, the higher the IBO is, i.e. the PA is away from its saturation point, the more performance increases. Thus, our blind DL-based signal receiver built by two coupled CV-NNs solutions guarantees performance close to the analytic one.

We have also tested our receiver with a 32-QAM received signal and a time-varying vehicular channel. Fig. 6 describes different blind approaches in the case of the vehicular time-varying channel $h_v$. We can observe the robustness of our approach even with a time-varying channel and different constellation orders.

<figure><img src="image_7.png" /><figcaption>FIGURE 6. BER versus SNR and IBO using $h_v$ in the case of a 32-QAM signal in a blind mode.</figcaption></figure>

## B. GENERIC BLIND DL-BASED SIGNAL RECEIVER

We trained CV-NN 1 with the multi-criteria cost function. For each equalized symbol, we apply the KNN algorithm for classification and finally, we update the cost function to reach one that matches the classified constellation. Once the CV-NN 1 is trained, we train CV-NN 2 with the resulting cost function obtained after the convergence of CV-NN1 without applying the AMC.

The sub-figures in Fig. 7 are simulated with a 16-QAM signal and a radio channel in a generic mode. The sub-figures in Fig. 8 are simulated with a 32-QAM signal and the time-varying vehicular channel in a generic mode. The obtained results confirm the performance of our contribution in a generic mode with different constellations orders and channels propagation.

Fig. 7(a) and Fig. 8(a) show the data source to be transmitted, Fig. 7(b) and Fig. 8(b) illustrate the data after the application of PA, Fig. 7(c) and Fig. 8(c) draw the received data at the channel output, these data are the receiver input and Fig. 7(d) and Fig. 8(d) plot the CV-NN 1 output signal constellation. We can see that the constellation sub-clusters are not distributed in the right places. Also, there are overlaps between the decision areas corresponding to the different symbols. Indeed CV-NN 1 has mitigated only ISI and not PANL, which will lead to higher BER values.

<figure><img src="image_8.png" /><figcaption>FIGURE 7. Different constellation for IBO = 4dB and SNR = 20dB using $h_r$ with 16-QAM signal in generic mode.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>FIGURE 8. Different constellation for IBO = 4dB and SNR = 20dB using $h_v$ with 32-QAM signal in generic mode.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>FIGURE 9. BER versus SNR and IBO using $h_r$ in the case of a 16-QAM signal in a generic mode.</figcaption></figure>

Fig. 7(e) and Fig. 8(e) plot CV-NN 2 output signal constellation. It can be noticed that the constellation sub-clusters are almost distributed at the proper positions. In addition, the decision areas corresponding to the different symbols are well separated. This is reflected in low BER values. This performance improvement is due to the ISI mitigation and PANL compensation by CV-NN 1 and CV-NN 2, respectively.
<figure><img src="image_11.png" /><figcaption>FIGURE 10. BER versus SNR and IBO using $h_v$ in the case of a 32-QAM signal in a generic mode.</figcaption></figure>

Fig.9 describes different generic approaches simulated with a 16-QM signal and the digital radio channel $h_r$. We can observe a slight performance degradation for all algorithms compared to Fig.5. This was expected since the receiver has no idea about the transmitted modulation and the cost function used for this case is generic.

In Fig.10, we present the performance of the generic DL-based signal receiver with a 32-QAM signal and the vehicular time-varying channel $h_v$. We noticed an interesting BER performance and robust convergence despite channel time variation during transmission.

## VI. CONCLUSION

This paper presents two contributions. The first one jointly and blindly corrects the effects of inter-symbol interference introduced by the propagation channel with the non-linearities introduced by power amplifiers without memory and presenting both amplitude and phase conversions. The second one jointly and generic corrects the same effects. Each of the two proposed non-linear receivers is a cascade of two neural networks, the first of which corrects the ISI and the second compensates the non-linearities. For the blind receiver, the two complex-valued neural networks are trained by employing the MSQD-$\ell_2$ criterion, and for the generic receiver the first CV-NN is trained using an updated multi-criteria cost function, and the second CV-NN is trained using the resulting cost function from the CV-NN 1. The numerical results show a good performance in terms of BER with a stationary channel and a time-varying vehicular channel. The performance achieved by the DL-based signal receiver motivates us to extend our work to multiple inputs multiple outputs (MIMO) communication systems in the presence of power amplifiers exhibiting memory effects.

## REFERENCES

[1] M. Jung, S. Min, and B.-W. Min, "A quasi-balanced power amplifier with feedforward linearization," *IEEE Microw. Wireless Compon. Lett.*, vol. 32, no. 4, pp. 312-315, Apr. 2022.

[2] T. Pengliang, "Feedback linearization of MIMO nonlinear system with measurable disturbance," in *Proc. 12th Int. Conf. Measuring Technol. Mechatronics Autom. (ICMTMA)*, Feb. 2020, pp. 744-749.

[3] H. Liu, X. Yang, P. Chen, M. Sun, B. Li, and C. Zhao, "Deep learning based nonlinear signal detection in millimeter-wave communications," *IEEE Access*, vol. 8, pp. 158883-158892, 2020.

[4] L. Xu, F. Gao, W. Zhang, and S. Ma, "Model aided deep learning based MIMO OFDM receiver with nonlinear power amplifiers," in *Proc. IEEE Wireless Commun. Netw. Conf. (WCNC)*, Mar. 2021, pp. 1-6.

[5] R. J. Thompson and X. Li, "Integrating Volterra series model and deep neural networks to equalize nonlinear power amplifiers," in *Proc. 53rd Annu. Conf. Inf. Sci. Syst. (CISS)*, Mar. 2019, pp. 1-6.

[6] N. D. Lahbib, M. Cherif, M. Hizem, and R. Bouallegue, "BER analysis and CS-based channel estimation and HPA nonlinearities compensation technique for massive MIMO system," *IEEE Access*, vol. 10, pp. 27899-27911, 2022.

[7] M. Noweir, M. Helaoui, D. Oblak, W. Chen, and F. M. Ghannouchi, "Linearization of radio-over-fiber cloud-RAN transmitters using pre- and post-distortion techniques," *IEEE Photon. Technol. Lett.*, vol. 33, no. 7, pp. 339-342, Apr. 1, 2021.

[8] K. Yuan, J. Zhuo, W. Gao, L. Zhang, and J. Chen, "Diffusion constant modulus algorithm for blind equalization," in *Proc. IEEE Int. Conf. Signal Process., Commun. Comput. (ICSPCC)*, Oct. 2022, pp. 1-6.

[9] J. Yang, J.-J. Werner, and G. A. Dumont, "The multimodulus blind equalization and its generalized algorithms," *IEEE J. Sel. Areas Commun.*, vol. 20, no. 5, pp. 997-1015, Jun. 2002.

[10] M. Lázaro, I. Santamaría, D. Erdogmus, K. E. Hild, C. Pantaleón, and J. C. Principe, "Stochastic blind equalization based on PDF fitting using Parzen estimator," *IEEE Trans. Signal Process.*, vol. 53, no. 2, pp. 696-704, Feb. 2005.

[11] S. Fki, M. Messai, A. Aïssa-El-Bey, and T. Chonavel, "Blind equalization based on PDF fitting and convergence analysis," *Signal Process.*, vol. 101, pp. 266-277, Aug. 2014.

[12] D. Wang and D. Wang, "Generalized derivation of neural network constant modulus algorithm for blind equalization," in *Proc. 5th Int. Conf. Wireless Commun., Netw. Mobile Comput.*, Sep. 2009, pp. 1-4.

[13] C. You and D. Hong, "Nonlinear blind equalization schemes using complex-valued multilayer feedforward neural networks," *IEEE Trans. Neural Netw.*, vol. 9, no. 6, pp. 1442-1455, Nov. 1998.

[14] C. Farhati, S. Fki, A. A. El Bey, and F. Abdelkefi, "Blind channel equalization based on complex-valued neural network and probability density fitting," in *Proc. Int. Wireless Commun. Mobile Comput. (IWCMC)*, 2022, pp. 773-777.

[15] E. K. Chemweno, "Adaptive channel equalization using multilayer perceptron neural networks with variable learning rate parameter," *IOSR J. VLSI Signal Process.*, vol. 7, no. 2, pp. 34-40, Apr. 2017.

[16] S.-C.-K. Kalla, C. Gagné, M. Zeng, and L. A. Rusch, "Recurrent neural networks achieving MLSE performance for optical channel equalization," *Opt. Exp.*, vol. 29, no. 9, pp. 13033-13047, 2021.

[17] A. Caciularu and D. Burshtein, "Unsupervised linear and nonlinear channel equalization and decoding using variational autoencoders," *IEEE Trans. Cognit. Commun. Netw.*, vol. 6, no. 3, pp. 1003-1018, Sep. 2020.

[18] S. Barbarossa, A. Swami, B. M. Sadler, and G. Spadafora, "Classification of digital constellations under unknown multipath propagation conditions," *Proc. SPIE*, vol. 4045, pp. 175-186, Jul. 2000.

[19] Q. Shi, "Blind equalization and characteristic function based robust modulation recognition," in *Proc. 14th Int. Conf. Adv. Commun. Technol. (ICACT)*, Feb. 2012, pp. 660-664.

[20] S. Fki, A. Aïssa-El-Bey, and T. Chonavel, "Blind equalization and automatic modulation classification based on PDF fitting," in *Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP)*, Apr. 2015, pp. 2989-2993.

[21] C. Farhati, S. Fki, A. Aïssa-El-Bey, and F. Abdelkefi, "Automatic modulation classification for multi-criteria generic channel equalization," in *Proc. 97th Veh. Technol. Conf. (VTC)*, 2023, pp. 2-4.

[22] A. K. Ali and E. Erçelebi, "An M-QAM signal modulation recognition algorithm in AWGN channel," *Sci. Program.*, vol. 2019, May 2019, Art. no. 6752694.

[23] I. A. Hashim, J. W. A. Sadah, T. R. Saeed, and J. K. Ali, "Recognition of QAM signals with low SNR using a combined threshold algorithm," *IETE J. Res.*, vol. 61, no. 1, pp. 65-71, Jan. 2015.

[24] P. Mukherjee, S. Lajnef, and I. Krikidis, "MIMO SWIPT systems with power amplifier nonlinearities and memory effects," *IEEE Wireless Commun. Lett.*, vol. 9, no. 12, pp. 2187-2191, Dec. 2020.
[25] A. Cheaito, M. Crussière, J.-F. Hélard, and Y. Louët, "Quantifying the memory effects of power amplifiers: EVM closed-form derivations of multicarrier signals," *IEEE Wireless Commun. Lett.*, vol. 6, no. 1, pp. 34–37, Feb. 2017.

[26] R. Zayani, H. Shaïek, and D. Roviras, "Efficient precoding for massive MIMO downlink under PA nonlinearities," *IEEE Commun. Lett.*, vol. 23, no. 9, pp. 1611–1615, Sep. 2019.

[27] K. Komatsu, Y. Miyaji, and H. Uehara, "Weighted least squares with orthonormal polynomials and numerical integration for estimation of memoryless nonlinearity," *IEEE Wireless Commun. Lett.*, vol. 9, no. 12, pp. 2197–2201, Dec. 2020.

[28] V. Tapio and M. Juntti, "Non-linear self-interference cancelation for full-duplex transceivers based on Hammerstein–Wiener model," *IEEE Commun. Lett.*, vol. 25, no. 11, pp. 3684–3688, Nov. 2021.

[29] M. Cherif, A. Arfaoui, and R. Bouallegue, "Autoencoder-based deep learning for massive multiple-input multiple-output uplink under high-power amplifier non-linearities," *IET Commun.*, vol. 17, no. 2, pp. 162–170, Jan. 2023.

[30] H. Zhang, X. Liu, D. Xu, and Y. Zhang, "Convergence analysis of fully complex backpropagation algorithm based on Wirtinger calculus," *Cognit. Neurodyn.*, vol. 8, no. 3, pp. 261–266, Jun. 2014.

[31] X. Cui, K. Liu, and Y. Liu, "Novel linear companding transform design based on linear curve fitting for PAPR reduction in OFDM systems," *IEEE Commun. Lett.*, vol. 25, no. 11, pp. 3604–3608, Nov. 2021.

[32] *Realistic Power Amplifier Model for the New Radio Evaluation*, document TSG-RAN WG4, Nokia, 3GPP, 2016.

[33] H. Shaiek, R. Zayani, Y. Medjahdi, and D. Roviras, "Analytical analysis of SER for beyond 5G post-OFDM waveforms in presence of high power amplifiers," *IEEE Access*, vol. 7, pp. 29441–29452, 2019.

[34] P. Marchand, C. Le Martret, and J.-L. Lacoume, "Classification of linear modulations by a combination of different orders cyclic cumulants," in *Proc. IEEE Signal Process. Workshop Higher-Order Statist.*, Jul. 1997, pp. 47–51.

[35] P. Marchand, J.-L. Lacoume, and C. Le Martret, "Multiple hypothesis modulation classification based on cyclic cumulants of different orders," in *Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP)*, vol. 4, May 1998, pp. 2157–2160.

[36] L. N. Trefethen, *Approximation Theory and Approximation Practice, Extended Edition*. Philadelphia, PA, USA: SIAM, 2019.

[37] S. Kojima, K. Maruta, and C.-J. Ahn, "Adaptive modulation and coding using neural network based SNR estimation," *IEEE Access*, vol. 7, pp. 183545–183553, 2019.

[38] D. Z. Rodríguez, R. L. Rosa, F. L. Almeida, G. Mittag, and S. Möller, "Speech quality assessment in wireless communications with MIMO systems using a parametric model," *IEEE Access*, vol. 7, pp. 35719–35730, 2019.

<figure><img src="image_12.png" /></figure>

**CHOUAIB FARHATI** received the engineering degree in computer science from the Faculty of Sciences of Tunis, in 2010. He is currently pursuing the Ph.D. degree in information technology and communications with the High School of Communications of Tunis, University of Carthage, Tunisia. His main research interests include modern wireless communications and coding, artificial intelligence, deep learning, and signal processing.

<figure><img src="image_13.png" /></figure>

**SOUHAILA FEKI** received the Diploma (engineering) degree in digital communication systems from the Institut Polytechnique de Bordeaux, France, the master's (research) degree in image and signal processing from the University of Bordeaux I, in 2011, and the Ph.D. degree from IMT Atlantique (Télécom Bretagne), Brest, France, in 2015. From 2015 to 2022, she was an Associate Professor with the Higher Institute of Applied Science and Technology of Sousse,

Tunisia. She is currently an Associate Professor with the National School of Advanced Science and Technology of Borj Cédria, Tunisia. Her research interests include digital communications, detection and estimation, blind system equalization, automatic modulation classification, and AI for digital signal processing. She was a member of MEDIATRON Laboratory.

<figure><img src="image_14.png" /></figure>

**HMAIED SHAIEK** (Member, IEEE) received the engineering degree from École Nationale d'Ingénieurs de Tunis, in 2002, the M.S. degree from Université de Bretagne Occidentale, in 2003, the Ph.D. degree from Telecom Bretagne-Brest, in 2007, and the HDR degree from the National Conservatory of Arts and Crafts (CNAM), in 2022. He was with Canon Inc., until 2009. In 2011, he joined CNAM as an Associate Professor in electronics and signal processing. His teaching

activities are in the fields of electronics, signal processing, and digital communications. He contributed to the FP7 EMPHATIC European project and is involved in four national projects. He has authored or coauthored three patents, 18 journal articles, one book chapter, and over 42 conference papers. His research interests include the performances analysis of multicarrier modulations with nonlinear power amplifiers, PAPR reduction, power amplifier linearization, RIS, and cell-free MIMO systems.

<figure><img src="image_15.png" /></figure>

**ALI DZIRI** (Member, IEEE) received the M.Eng. degree in electrical engineering and the master's degree in signal processing for communications from Ecole Nationale d'Ingénieurs de Tunis (ENIT), in 1997 and 1998, respectively, and the Ph.D. degree in signal processing for communications from the National Conservatory of Arts and Crafts (CNAM), Paris, in May 2004. From 2005 to 2006, he was a contract Lecturer (ATER) with CNAM. In 2007, he joined as a

Postdoctoral Researcher with Supélec, Paris, in the collaboration with Alcatel-Lucent, for two years. Since 2009, he has been a Senior Research Engineer with the Centre d'Etudes et De Recherche en Informatique et Communications (CEDRIC Laboratory). He contributed and involved in more than ten European and national projects. He has authored or coauthored more than 30 journal and conference papers. His research interests include cooperative and relay-assisted communications for 5G and beyond, RIS-based communication systems, statistical characterization and modeling of MIMO fading channels, cross layer approach for wireless sensor networks, software-defined radio (SDR), and video (H264 AVC, H264 SVC) compression and transmission.

<figure><img src="image_16.png" /></figure>

**ABDELDJALIL AÏSSA EL BEY** (Senior Member, IEEE) received the state engineering degree from École Nationale Polytechnique (ENP), Algiers, Algeria, in 2003, the M.S. degree in signal processing from Supélec and Paris XI University, Orsay, France, in 2004, and the Ph.D. degree in signal and image processing from École Nationale Supérieure des Télécommunications (ENST), Paris, France, in 2007. In 2007, he joined the Department of Signal and Communications,

IMT Atlantique (Télécom Bretagne), Brest, France, as an Associate Professor, and has been a Full Professor, since 2015. He was a Visiting Researcher with Fujitsu Laboratories, Japan, and the Department of Electrical and Electronic Engineering, The University of Melbourne, Australia, in 2010 and 2015, respectively. His research interests include blind source separation, blind system identification and equalization, compressed sensing, sparse signal processing, statistical signal processing, wireless communications, and adaptive filtering.

**FATMA ABDELKEFI** (Senior Member, IEEE) received the degree in electrical engineering from École Nationale d'Ingénieurs de Tunis, Tunis, Tunisia, in 1998, and the M.S. degree in digital communications systems and the Ph.D. degree (Hons.) from École Nationale Supérieure des Télécommunications, Paris, France, in 1999 and 2002, respectively. Since 2009, she has been with the High School of Communications of Tunis, University of Carthage, Tunis, where she is currently a Professor. Her research interests include digital communications, information theory, detection and estimation, equalization, multicarrier systems, compressive sensing, and cognitive radio.