# Theoretical analysis and compensation for the joint effects of HPA nonlinearity and RF crosstalk in VBLAST MIMO-OFDM systems over Rayleigh fading channel

Maha Cherif Dakhli, Rafik Zayani, Oussama Belkacem, Ridha Bouallegue

► To cite this version:

Maha Cherif Dakhli, Rafik Zayani, Oussama Belkacem, Ridha Bouallegue. Theoretical analysis and compensation for the joint effects of HPA nonlinearity and RF crosstalk in VBLAST MIMO-OFDM systems over Rayleigh fading channel. EURASIP Journal on Wireless Communications and Networking, 2014, 2014 (1), 10.1186/1687-1499-2014-61. hal-02454870

HAL Id: hal-02454870

https://hal.science/hal-02454870

Submitted on 9 Mar 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
RESEARCH

Open Access

# Theoretical analysis and compensation for the joint effects of HPA nonlinearity and RF crosstalk in VBLAST MIMO-OFDM systems over Rayleigh fading channel

Maha Cherif Dakhli*, Rafik Zayani, Oussama B Belkacem and Ridha Bouallegue

**Abstract**

In this paper, we investigate the joint effects of nonlinear distortions due to high-power amplifier (HPA) and radio-frequency (RF) crosstalk on the performance of a multiple-input multiple-output-orthogonal frequency division multiplexing (MIMO-OFDM) system. The performance of the vertical bell laboratories layered space-time (VBLAST) MIMO-OFDM system, in terms of bit error rate (BER) and system capacity, is derived and evaluated for a Rayleigh flat fading channel. In the performance analysis, M-ary quadrature amplitude modulation (M-QAM) is considered and the joint effects of nonlinear HPA distortions and RF crosstalk are studied. Theoretical results and comparisons are provided for several system parameters, such as the input back-off (IBO) and crosstalk intensity. It is shown that the results obtained from the theoretical study are very close to the ones obtained from the simulation. Indeed, we introduce a nonlinear distortion cancellation (NDC) technique to estimate and mitigate nonlinear distortions and crosstalk effects jointly.

**Keywords:** MIMO; OFDM; HPA; Crosstalk; Nonlinear distortion cancellation; MMSE

**1. Introduction**

The combination of multiple-input multiple-output (MIMO) and orthogonal frequency division multiplexing (OFDM) guarantees high performance with high data rates in 4G broadband wireless communications [1]. Indeed, OFDM combats frequency-selective fading effectively by dividing a wideband frequency-selective fading channel into parallel narrowband flat fading subchannels, while MIMO systems increase the data rate significantly over single-input single-output (SISO) systems by employing multiple transmit and receive antennas.

Like in classical SISO-OFDM, MIMO-OFDM exhibits large peak-to-average power ratios (PAPR), i.e., large fluctuations in their signal envelopes. Moreover, the performance of the transceiver is very sensitive to nonlinear distortions caused by high-power amplifiers (HPAs). On the other hand, MIMO transceivers use multiple transmission/reception paths implemented in the same chipset.

Crosstalk or coupling effect is the result of the interference between the signals of different paths, and it affects the quality of the transmitted signal and degrades the system performance. This type of crosstalk can be linear or nonlinear as explained in [2], where the authors discussed the difference between linear MIMO crosstalk, which is relatively benign since it is being corrected by the MIMO equalizer, and nonlinear crosstalk that is generated before the HPA. This nonlinear crosstalk is not corrected by the MIMO receiver, and it is thus much more harmful. HPA nonlinearity in SISO-OFDM systems was investigated in [3], where the output at the HPA is expressed as the summation of the input signal multiplied by a complex scale factor, and an additive Gaussian noise which is uncorrelated with the input signal.

Recent research effort has dealt with the issue of HPA nonlinearity in MIMO systems. For instance, based on this study, the effect of HPA nonlinearity on the bit error rate of quadrature amplitude modulation (QAM) was analyzed in [4] for MIMO systems. In addition, the impact of HPA nonlinearity on the bit error rate was studied in [5] for

<footnote>* Correspondence: maha.cherif@gmail.com
Innov'COM Lab, Sup'Com, Carthage University, Ariana 2083, Tunisia</footnote>
SDMA-OFDM systems and was analyzed in [3] in order to determine the parameters of amplification such as the complex attenuation coefficient $K_0$ and the variance of the nonlinear distortion $\sigma_d^2$.

Up to now, in the open literature, these terms have been investigated analytically only for the soft envelope limiter (SEL) model [3], and a similar framework was proposed for the traveling wave tube amplifier (TWTA) and the solid-state power amplifier (SSPA) models, but limited actually to a semi-analytic study [3,5].

The compensation methods and comprehensive theoretical analysis of several radio-frequency (RF) impairments together, in MIMO communication systems, are extremely challenging. A crossover digital predistorter was proposed for the compensation of HPA nonlinearity and crosstalk in MIMO systems and was evaluated through simulations [6,7].

In this paper, we investigate the joint effects of HPA nonlinearity and RF crosstalk in vertical bell laboratories layered space-time (VBLAST) MIMO-OFDM systems, and we derive the new closed-form probability density function (pdf) signal-to-noise ratio (SNR) in different scenarios: (1) nonlinear MIMO-OFDM system and (2) nonlinear MIMO-OFDM system with crosstalk.

In addition, we introduce a new approach to derive analytically $K_0$ and $\sigma_d^2$, and this approach is valid for any HPA model, such as TWTA, SSPA, and the real measured one. Thus, a polynomial approximation is considered in order to calculate the values of these parameters. Based on this approximation, we provide an analytical performance evaluation, in terms of bit error rate (BER) and channel capacity, of a VBLAST MIMO-OFDM system under a Rayleigh fading channel and memoryless nonlinear HPA model. Finally, we introduce a compensation method that aims to estimate and mitigate simultaneously the HPA nonlinearity and RF crosstalk at the receiver side. This proposed method is based on the theoretical analysis done in this paper.

Results were supported with MATLAB simulation of a VBLAST MIMO-OFDM system under a Rayleigh fading channel. The remainder of this paper is organized as follows: In Section 2, the system model is introduced. The approximation of the HPA model is studied in Section 3. The performance of the VBLAST MIMO-OFDM system in terms of BER and capacity is evaluated in Section 4. Section 5 presents a nonlinear distortion cancellation (NDC) technique for compensating the nonlinear crosstalk. Numerical results and discussions are presented in Section 6, while the conclusion is given in Section 7.

## 2. System model

Let us consider the baseband equivalent VBLAST MIMO-OFDM system with $Nt$ transmit antennas and $Nr$ receive antennas shown in Figure 1. It is assumed that the system employs $Nc$ subcarriers and M-QAM, and the fast Fourier transform (FFT) matrix is denoted by $V \in \mathbb{C}^{Nc \times Nc}$.

The frequency domain (FD) symbol $X_k$ is assumed to contain the source information, is mapped by using a VBLAST encoder, and is transmitted over a Rayleigh fading channel. The transmitted signal $x_k(n)$ from $k$ antenna at time instant $n$ is obtained by means of an inverse fast Fourier transform (IFFT) of the $X_k(n)$ and is given by

$$x_k(n) = V^H X_k(n) \quad (1)$$

### 2.1 Linear MIMO-OFDM system

Before reaching the $Nr$ receiving antennas, the transmitted signals are affected by the propagation channel, which can be modeled by the propagation matrix $H_{m,k}$. The received signal, in a linear HPA case, is represented by

$$\mathbf{y}_m^L(n) = \sum_{k=1}^{Nt} \mathbf{H}_{m,k}(n)\mathbf{x}_k(n) + \mathbf{n}_m(n) \quad (2)$$

where $\mathbf{n}_m(n)$ is the additive white Gaussian noise (AWGN) and $\mathbf{H}_{m,k}(n)$ is an $Nc \times Nc$ circulant time domain (TD) channel matrix at time instant $n$, which is formed by the channel response vector $\mathbf{h}_{m,k}(n)$ for the link between transmit antenna $k$ and receive antenna $m$. The FD expression of the received signal is obtained by taking the FFT of Equation 2 giving

$$\mathbf{Y}_m^L(n) = V \mathbf{y}_m^L(n) \quad (3)$$

Let $\mathbf{Y}^L(n,i) = V\mathbf{y}^L(n,i) = [Y_1(n,i),..., Y_{Nr}(n,i)]^T$ denote the vector of received signals in the linear case for all antenna on subcarrier $i$. Then, the received signal vector for each subcarrier can be expressed as

$$\mathbf{Y}^L(n, i) = \mathbf{H}(n, i)\mathbf{X}(n, i) + \mathcal{N}(n, i) \quad (4)$$

where $\mathbf{X}(n,i) = [X_1(n,i),..., X_{Nr}(n,i)]^T$ is the vector that contains transmitted signals from each antenna and the Gaussian noise term $\mathcal{N}(n,i)$ as obtained by FFT of the thermal Gaussian noise of the receiver has zero mean and variance $\sigma_n^2$. The frequency domain channel transfer matrix $\mathbf{H}(n,i)$ in Equation 4 is given by

$$\mathbf{H}(n, i) = \begin{bmatrix} h_{1,1}(n, i) & h_{1,2}(n, i) & \cdots & h_{1,Nt}(n, i) \\ h_{2,1}(n, i) & h_{2,2}(n, i) & \cdots & h_{2,Nt}(n, i) \\ \vdots & \vdots & \ddots & \vdots \\ h_{Nr,1}(n, i) & h_{Nr,2}(n, i) & \cdots & h_{Nr,Nt}(n, i) \end{bmatrix} \quad (5)$$

where $h_{m,k}(n,i)$ denotes the $Nr \times Nt$ channel coefficient on subcarrier $i$ at time $n$ between the $k$th transmit and $m$th receive antennas.
<figure><img src="image_2.png" /><figcaption>Figure 1 VBLAST MIMO-OFDM system with nonlinear HPA.</figcaption></figure>

## 2.2 Nonlinear MIMO-OFDM system

### 2.2.1 Nonlinear HPA model

Memoryless power amplifiers are completely characterized by their amplitude (AM/AM) and phase (AM/PM) conversions, which depend only on the current input signal value. Considering the complex envelope $x_k = \rho_k e^{j\theta_k}$ of the HPA input at each branch where $\rho_k$ and $\theta_k$ denote, respectively, the amplitude and the phase of $x_k$, the complex envelope of a memoryless HPA output $z_k(n)$ can be modeled as follows:

$$ 
\begin{aligned}
z_k(n) &= g[x_k(n)] = A(\rho_k)e^{j(\theta_k + P(\rho_k))} \\
&= S(\rho_k)e^{j\theta_k}
\end{aligned} 
\quad (6) $$

where $g[\cdot]$ is the HPA transfer function and $A(\cdot)$ and $P(\cdot)$ denote the HPA AM/AM and AM/PM conversions, respectively. For a nonlinear HPA model, we consider Saleh's model [8], which has the advantage of good accuracy and reasonable complexity than other models. The AM/AM and AM/PM conversions can be represented as follows [5,9]:

$$ A(\rho_k) = \frac{\alpha_a \rho_k}{1 + \beta_a \rho_k^2}, \quad P(\rho_k) = \frac{\alpha_p \rho_k^2}{1 + \beta_p \rho_k^2} \quad (7) $$

where $\alpha_a$ and $\beta_a$ are the parameters that decide the nonlinear level, and $\alpha_p$ and $\beta_p$ are phase displacements. For validation of our theoretical results, the operating point of the amplifier is usually identified by the 'back-off'. In the simulations, we define the input back-off as [9]

$$ IBO = 10 \log_{10} \left( \frac{A_0^2}{P_0} \right) \quad (8) $$

where $A_0$ is the saturation amplitude at the input of HPA and $P_0$ is the input average power. From the Bussgang theorem and by extending that to complex Gaussian processes, the output signal at the HPA can be expressed as [3]

$$ z_k(n) = K_0 x_k(n) + d_k(n) \quad (9) $$

where $K_0$ is the attenuation coefficient and $d_k(n)$ is the nonlinear distortion term uncorrelated with the input

signal $x_k(n)$. The value of $K_0$ is given by Dardari et al. in ([3], eq. (19)) by

$$ K_0 = \frac{1}{2} E \left[ S'(\rho_k) + \frac{S(\rho_k)}{\rho_k} \right] \quad (10) $$

where $S'(\rho_k)$ denotes the differential of $S(\rho_k)$. Furthermore, the variance of the nonlinear distortion term ($d_k(n)$) is given by ([3], eq. (37))

$$ 
\begin{aligned}
\sigma_d^2 &= E[|\mathbf{d}_k(n)|^2] \\
&= E[|\mathbf{z}_k(n)|^2] - |K_0|^2 E[|\mathbf{x}_k(n)|^2] \\
&= E[|S(\rho_k)|^2] - |K_0|^2 E[|\rho_k|^2]
\end{aligned} 
\quad (11) $$

The received signal at each antenna $\mathbf{y}_m^{\text{NL}}(n)$ is constituted by the superposition of independently faded signals, associated with the $Nt$ antennas sharing the same space-frequency resource. The received signal is assumed to be corrupted by a Gaussian noise at the array elements, and it is given by

$$ \mathbf{y}_m^{\text{NL}}(n) = \sum_{k=1}^{Nt} \mathbf{H}_{m,k}(n)(K_0 \mathbf{x}_k(n) + \mathbf{d}_k(n)) + \mathbf{n}_m(n) \quad (12) $$

The frequency domain expression by taking the FFT of the received signal is given by

$$ \mathbf{Y}_m^{\text{NL}}(n) = V \mathbf{y}_m^{\text{NL}}(n) \quad (13) $$

The received signal vector at each antenna on subcarrier (i) can be written as [5,10]

$$ \mathbf{Y}^{\text{NL}}(n, i) = K_0 \mathbf{H}(n, i) \mathbf{X}(n, i) + \mathbf{H}(n, i) \mathbf{D}(n, i) + \mathcal{N}(n, i) \quad (14) $$

### 2.2.2 Nonlinear MIMO-OFDM with RF crosstalk

In MIMO transceivers, crosstalk or coupling effects are the result of the interference between the signals of different paths [6,11]. It is worth noting here that this crosstalk would be more significant in a highly integrated circuit (IC) design, mainly when the size of the prototype is small. This undesired MIMO crosstalk can occur in the radio channel, between the antennas, and also in the radio IC. In [12], crosstalk between the different RF transceiver
chains has been identified as a key issue when integrating a MIMO radio IC. This type of crosstalk can be linear and nonlinear as explained in [2], where the authors discussed the difference between linear MIMO crosstalk, which is relatively benign since it is being corrected by the MIMO equalizer, and nonlinear crosstalk that is not corrected by the MIMO receiver and is thus much more harmful. In this paper, we focus on nonlinear crosstalk that occurs at the radio IC level and introduced before nonlinear HPA (see Figure 2).

The HPA's output with the effect of crosstalk can be modeled as

$$ 
\begin{aligned}
z_k(n) &= g\left[\mathbf{x}_k(n) + \sum_{l=1, l \neq k}^{Nt} \alpha_{lk}(n) \otimes \mathbf{x}_l(n)\right] \\
&= K_0 \mathbf{x}_k(n) + K_0 \sum_{l=1, l \neq k}^{Nt} \alpha_{lk}(n) \otimes \mathbf{x}_l(n) \\
&\quad + \mathbf{d}_k(n)
\end{aligned} 
\quad (15) $$

where $g[\cdot]$ is the HPA response for each branch, $\mathbf{x}_l(n)$ is the output of the FFT bloc of the $l$ path of the transceiver, and $\alpha_{lk}(n)$ is the filter representing the crosstalk and modeling the coupling between path $l$ to path $k$ [7]. Since the signals in different paths use the same operating frequency and have equal transmission power, crosstalk is more likely between the paths. In fact, the received signal of a nonlinear crosstalk is represented by

$$ 
\begin{aligned}
\mathbf{y}_m^{\text{NL+cross}}(n) = & \sum_{k=1}^{Nt} \mathbf{H}_{m,k}(n) \\
& \times \left( K_0 \mathbf{x}_k(n) + K_0 \sum_{l=1, l \neq k}^{Nt} \alpha_{lk}(n) \otimes \mathbf{x}_l(n) \right) \\
& + \mathbf{d}_k(n) + \mathbf{n}_m(n)
\end{aligned} 
\quad (16) $$

In order to simplify the presentation of the theoretical study, we limit the calculation for two transmit antennas and two receive antennas, and we consider that the

coefficients of the crosstalk are equal ($\alpha_{21} = \alpha_{12} = \alpha$) [6]. The frequency domain expression of the received signal vector for each subcarrier can be written as

$$ 
\begin{aligned}
\mathbf{Y}^{\text{NL+cross}}(n, i) = & K_0 \mathbf{H}(n, i) \mathbf{X}(n, i) \\
& + K_0 \alpha \mathbf{H}(n, i) \mathbf{X}(n, i) \\
& + \mathbf{H}(n, i) \mathbf{D}(n, i) + \mathcal{N}(n, i)
\end{aligned} 
\quad (17) $$

where $\mathbf{X}(n, i) = \begin{bmatrix} \mathbf{x}_2(n, i) \\ \mathbf{x}_1(n, i) \end{bmatrix}$.

## 2.3 MMSE receiver

The transmitted signals by different antennas on subcarrier $i$ can be estimated with the aid of a suitable linear combiner $\mathbf{W} \in \mathbb{C}^{N_r \times N_t}$. The minimum mean square error (MMSE) detection minimizes the mean square error between the transmit signal and the estimated signal [13-15]. The optimal MMSE weight is obtained as follows:

$$ 
\mathbf{W}_{(\text{MMSE})} = [\mathbf{H}^H(n, i)\mathbf{H}(n, i) + \sigma_n^2\mathbf{I}]^{-1}\mathbf{H}^H(n, i) \quad (18) $$

where $\sigma_n^2$ is the noise variance and $\mathbf{I}$ is the matrix identity. After equalization, the approximated signal can be expressed as

$$ 
\hat{\mathbf{X}}(n, i) = \mathbf{W}_{(\text{MMSE})} \mathbf{Y}(n, i) \quad (19) $$

## 3. Nonlinear distortion characterization

To evaluate the nonlinear distortion effects in MIMO-OFDM systems, in this section, we propose an analytical study to calculate the parameters of a power amplifier.

<figure><img src="image_3.png" /><figcaption>Figure 2 VBLAST MIMO-OFDM system with nonlinear crosstalk for $Nt = Nr = 2$.</figcaption></figure>
In [3,5,16], the terms of the nonlinear distortion are evaluated with semi-analytic expressions. Motivated by these results, we will expand this study in order to introduce a new approach to derivate analytically the complex attenuation coefficient $K_0$ and the variance of the nonlinear distortion $\sigma_d^2$, and this approach is valid for any HPA model even a real measured one. Thus, we will consider a polynomial approximation in order to calculate the values of these parameters.

### 3.1 Polynomial approximation of the nonlinear distortion

As it is expressed by Equations 10 and 11, the analytical computation of $K_0$ and the variance $\sigma_d^2$ is not feasible because we have to calculate the expectation ($E[.]$) of the complicated function. Therefore, we propose a new approach, which is based on polynomial approximation in order to estimate the nonlinear HPA. In this case, the output of HPA can be represented by

$$ 
\begin{aligned}
z(t) &= \sum_{l=0}^{L} p_{2l+1} x(t) |x(t)|^{2l} \\
&= e^{i\theta(t)} \sum_{l=0}^{L} p_{2l+1} \rho^{2l+1} = S(\rho(t)) e^{i\theta(t)}
\end{aligned} 
\quad (20) 
$$

where $L$ is the order of the polynomial and $p_{2l+1}$ are the complex coefficients. Therefore, the new expression of $S(\rho)$ is given by:

$$ S(\rho) = \sum_{l=0}^{L} p_{2l+1} \rho^{2l+1} \quad (21) $$

Then,

$$ S'(\rho) = \sum_{l=0}^{L} (2l + 1) p_{2l+1} \rho^{2l} \quad (22) $$

and

$$ \frac{S(\rho)}{\rho} = \sum_{l=0}^{L} p_{2l+1} \rho^{2l} \quad (23) $$

### 3.2 Estimation of $p_{2l+1}$

In order to estimate the polynomial coefficient $p_{2l+1}$, we use the least square (LS) approach [17]:

$$ \hat{p}_k = (X_k^H X_k)^{-1} X_k^H z_k \quad (24) $$

Figures 3 and 4 show the AM/AM and AM/PM approximation, respectively, using a polynomial with order $L = 10$. The considered HPA is Saleh's model with $\alpha_a = 1$,

$\beta_a = 1$, $\alpha_p = \frac{\pi}{3}$, and $\beta_p = 1$ and input back-off (IBO) of 8 dB. We note from these results the AM/AM and AM/PM characteristics given by Saleh's model and those obtained by polynomial approximation are well matched.

### 3.3 Analytical computation of $K_0$ and $\sigma_d^2$

Using this polynomial approximation, the analytical expression of $K_0$ is obtained by substituting (22) and (23) in (10) and we obtain

$$ 
\begin{aligned}
K_0 &= \frac{1}{2} E \left[ \sum_{l=0}^{L} (2l + 2) p_{2l+1} \rho^{2l} \right] \\
&= \frac{1}{2} \sum_{l=0}^{L} (2l + 2) p_{2l+1} E[\rho^{2l}]
\end{aligned} 
\quad (25) 
$$

This will be achieved by computing the 2 $l$-order moment ($E[\rho^{2l}]$), where $\rho$ can be written as

$$ \rho = (a^2 + b^2)^{1/2} \quad (26) $$

where $a$ and $b$ represent the real and imaginary parts, respectively. Then,

$$ 
\begin{aligned}
E[\rho^{2l}] &= E\left[(a^2 + b^2)^l\right] = E\left[\sum_{j=0}^{l} C_l^j a^{2j} b^{2l-2j}\right] \\
&= \sum_{j=0}^{l} C_l^j E[a^{2j}] E[b^{2l-2j}]
\end{aligned} 
\quad (27) 
$$

where

$$ C_l^j = \frac{l!}{j!(l-j)!} \quad (28) $$

$$ E[a^{2j}] = \sigma_a^{2j} (2j-1)!! \sigma_a^{2j} \frac{2^j \Gamma(\frac{2j+1}{2})}{\sqrt{\pi}} \quad (29) $$

$$ 
\begin{aligned}
E[b^{2l-2j}] &= \sigma_b^{2j} (2l-2j-1)!! \\
&= \sigma_b^{2l-2j} \frac{2^{l-j} \Gamma(\frac{2l-2j+1}{2})}{\sqrt{\pi}}
\end{aligned} 
\quad (30) 
$$

where !! represents the double factorial. Thus, $E[\rho^{2l}]$ is given by
<figure><img src="image_4.png" /><figcaption>Figure 3 AM/AM polynomial approximation.</figcaption></figure>

$$E[\rho^{2l}] = \sum_{j=0}^{l} \frac{l!}{j!(l-j)!} \sigma_a^{2j} \sigma_b^{2l-2j} \frac{2^j \Gamma(\frac{2j+1}{2})}{\sqrt{\pi}} \frac{2^{l-j} \Gamma(\frac{2l-2j+1}{2})}{\sqrt{\pi}} \quad (31)$$

where Γ is the gamma function, which is defined as

$$\Gamma(x) = \int_{0}^{+\infty} t^{x-1} e^{-t} dt \quad (32)$$

Finally, substituting (31) into (25), we obtain the following expression for the complex attenuation coefficient $K_0$:

$$K_0 = \frac{1}{2} \sum_{l=0}^{L} (2l+2)p_{2l+1} \times \left\{ \sum_{j=0}^{l} \frac{l!}{j!(l-j)!} \sigma_a^{2j} \sigma_b^{2l-2j} \frac{2^l \Gamma(\frac{2j+1}{2})}{\sqrt{\pi}} \frac{2^{l-j} \Gamma(\frac{2l-2j+1}{2})}{\sqrt{\pi}} \right\} \quad (33)$$

To compute the variance of the nonlinear distortion $\sigma_d^2$, we substitute (21) into (11), and the theoretical expression of $\sigma_d^2$ is given by

$$\begin{aligned} \sigma_d^2 = & \sum_{l=0}^{L} |p_{2l+1}|^2 E[\rho^{2(2l+1)}] \\ & + 2 \sum_{j,l=0, j \neq l}^{L} \Re e(p_{2l+1} p_{2j+1}) \\ & \times E[\rho^{(2l+1)+(2j+1)}] - |K_0|^2 E[\rho^2] \end{aligned} \quad (34)$$

where

$$\begin{aligned} E[\rho^2] &= \sigma_a^2 \frac{\Gamma(\frac{1}{2})}{\sqrt{\pi}} \frac{2\Gamma(\frac{3}{2})}{\sqrt{\pi}} + \sigma_b^2 \frac{2\Gamma(\frac{1}{2})}{\sqrt{\pi}} \frac{\Gamma(\frac{3}{2})}{\sqrt{\pi}} \\ &= \frac{2\Gamma(\frac{1}{2})\Gamma(\frac{3}{2})}{\pi} (\sigma_a^2 + \sigma_b^2) = \sigma_a^2 + \sigma_b^2 \end{aligned} \quad (35)$$

$$\begin{aligned} E[\rho^{2(2l+1)}] &= \sum_{j=0}^{2l+1} \frac{(2l+1)!}{j!(2l+1-j)!} \sigma_a^{2j} \\ & \qquad \sigma_b^{2(2l+1)-2j} \frac{2l\Gamma(\frac{2j+1}{2})}{\sqrt{\pi}} \frac{2^{2l+1-j}\Gamma(\frac{2(2l+1)-2j+1}{2})}{\sqrt{\pi}} \end{aligned} \quad (36)$$

$$\begin{aligned} E[\rho^{(2l+1)+(2j+1)}] &= \sum_{j=0}^{\frac{(2l+1)+(2j+1)}{2}} \frac{\frac{(2l+1)+(2j+1)}{2}!}{j!\left(\frac{(2l+1)+(2j+1)}{2}-j\right)!} \sigma_a^{2j} \\ & \qquad + \sigma_b^{(2l+1)+(2j+1)-2j} \frac{2l\Gamma(\frac{2j+1}{2})}{\sqrt{\pi}} \frac{2^{\frac{(2l+1)+(2j+1)}{2}-j}\Gamma(\frac{(2l+1)+(2j+1)-2j+1}{2})}{\sqrt{\pi}} \\ &= \sum_{j=0}^{l+j+1} \frac{(l+j+1)!}{j!(l+1)!} \sigma_a^{2j} \sigma_b^{2(l+1)} \frac{2l\Gamma(\frac{2j+1}{2})}{\sqrt{\pi}} \frac{2^{(l+1)}\Gamma(\frac{2l+3}{2})}{\sqrt{\pi}} \end{aligned} \quad (37)$$
Substituting (35), (36), and (37) in (34), we can rewrite
(34) as

$$
\sigma_d^2 = \sum_{l=0}^{L} |p_{2l+1}|^2 \left\{ \sum_{j=0}^{2l+1} \frac{(2l+1)!}{j!(2l+1-j)!} \sigma_a^{2j} \sigma_b^{2(2l+1)-2j} \frac{2^j \Gamma(\frac{2j+1}{2}) 2^{(2l+1)-j} \Gamma(\frac{2(2l+1)-2j+1}{2})}{\sqrt{\pi} \sqrt{\pi}} \right\} \\
+ 2 \sum_{j,l=0}^{L} \sum_{i,j=0}^{L} \Re e(p_{2l+1} p_{2j+1}) \left\{ \sum_{j=0}^{l+j+1} \frac{(l+j+1)!}{j!(l+1)!} \sigma_a^{2j} \sigma_b^{2(l+1)} \frac{2^j \Gamma(\frac{2j+1}{2}) 2^{(l+1)} \Gamma(\frac{2l+3}{2})}{\sqrt{\pi} \sqrt{\pi}} \right\} - |K_0|^2 (\sigma_a^2 + \sigma_b^2) \quad (38)
$$

Table 1 presents the comparison of simulation and
analytical results of K₀ and σ²ₐ obtained, respectively, by
((10) and (11)) and ((33) and (38)) for different values of
IBO. According to this table, we observe a good agree-
ment between simulation and theoretical results of the
both K₀ and σ²ₐ, which validate our analysis.

4. VBLAST MIMO-OFDM performance analysis

In this section, we evaluate analytically the performance
of VBLAST MIMO-OFDM systems in terms of BER and
the system capacity. The considered system uses M-QAM
under a Rayleigh flat fading channel.

4.1 BER performance

A fading channel can be considered as an AWGN with a
variable gain. The gain itself is considered as a random

variable with a given probability density function (pdf).
Moreover, the average BER can be calculated by aver-
aging BER for instantaneous SNR over its distribution:

$$
\text{BER} = \int_{0}^{+\infty} \text{BER}_{\text{M-QAM}}(\gamma) p_{\gamma}(\gamma) d\gamma \quad (39)
$$

where $p_{\gamma}(\gamma)$ is the pdf of the instantaneous SNR ($\gamma$).

On the other hand, the bit error rate of a system,
which implements M-QAM over a Rayleigh fading chan-
nel, can be defined as [18]

$$
\mathrm{BER}_{\mathrm{M-QAM}} = \frac{4}{\log_2 M} Q\left(\sqrt{\frac{3\gamma_b \log_2 M}{M-1}}\right) \quad (40)
$$

where *M* denotes the modulation order and
*γ*<sub>b</sub> = σ<sub>a</sub><sup>2</sup>/σ<sub>n</sub><sup>2</sup>
while *Q* is the standard *Q*-function defined as [18]

<figure><img src="image_5.png" /></figure>
<table><caption>**Table 1 Comparison of analytical and simulation results of K₀ and σ²ₐ for different values of IBO, L = 10, and Saleh's model**</caption><thead><tr><th rowspan="2">IBO (dB)</th><th colspan="2">Simulation</th><th colspan="2">Analytical method</th></tr><tr><th>K₀</th><th>σ²ₐ</th><th>K₀</th><th>σ²ₐ</th></tr></thead><tbody><tr><td>4</td><td>0.5556 + 0.2152i</td><td>0.0142</td><td>0.5560 + 0.2152i</td><td>0.0142</td></tr><tr><td>6</td><td>0.6625 + 0.1953i</td><td>0.0074</td><td>0.6627 + 0.1953i</td><td>0.0074</td></tr><tr><td>8</td><td>0.7556 + 0.1643i</td><td>0.0034</td><td>0.7560 + 0.1641i</td><td>0.0033</td></tr><tr><td>10</td><td>0.8304 + 0.1287i</td><td>0.0013</td><td>0.8302 + 0.1288i</td><td>0.0013</td></tr></tbody></table>

$$Q(x) = \frac{1}{\sqrt{2\pi}} \int_{\chi}^{+\infty} e^{-\frac{x^2}{2}} dx \quad (41)$$

In this investigation, we calculate, theoretically, the instantaneous SNR (y) and its pdf (p<sub>y</sub>(y)) in different scenarios such as linear MIMO-OFDM, nonlinear MIMO-OFDM, and nonlinear MIMO-OFDM with crosstalk. This analysis will be used to calculate the average BER.

### 4.1.1 Linear MIMO-OFDM system

Assuming perfect channel equalization in the receiver, a soft estimate can be obtained by applying the MMSE equalizer that's given by

$$\hat{\mathbf{X}}(n,i) = [\mathbf{H}^H(n,i)\mathbf{H}(n,i) + \sigma_n^2\mathbf{I}]^{-1}\mathbf{H}^H(n,i) \times [\mathbf{H}(n,i)\mathbf{x}(n,i) + \mathcal{N}(n,i)] \quad (42)$$

The instantaneous SNR in the linear case after equalization is

$$\gamma_L = \frac{|[\mathbf{H}^H(n,i)\mathbf{H}(n,i) + \sigma_n^2\mathbf{I}]^{-1}\mathbf{H}^H(n,i)\mathbf{H}(n,i)|^2 \sigma_n^2}{|[\mathbf{H}^H(n,i)\mathbf{H}(n,i) + \sigma_n^2\mathbf{I}]^{-1}\mathbf{H}^H(n,i)|^2 \sigma_n^2} \\ = \frac{\sigma_n^2 |\mathbf{H}(n,i)|^2}{\sigma_n^2} = \gamma_b |\mathbf{H}(n,i)|^2 \quad (43)$$

where $\mathbf{H}(n,i)$ is assumed to be a flat fading one, and it is exponentially distributed with a pdf given by

$$p_{\gamma}(\gamma) = \frac{1}{\Omega} e^{-\frac{\gamma}{\Omega}} \quad (44)$$

where $\Omega = E[|\mathbf{H}(n,i)|^2]$.

Then, by substituting (40) and (44) into (39), the average BER of the M-QAM MIMO-OFDM system can be expressed as

$$\text{BER}^{\text{L}} = \int_{0}^{+\infty} \frac{4}{\log_2 M} Q\left(\sqrt{\frac{3\gamma_b \log_2 M}{M-1}}\right) \frac{1}{\Omega} e^{-\frac{\gamma}{\Omega}} d\gamma \quad (45)$$

As shown in [18,19],

$$\text{BER}^{\text{L}} = \frac{(M-1)}{M \log_2 M} \left[ 1 - \frac{1}{\sqrt{1 + \frac{M^2-1}{3\gamma_b \log_2 M}}} \right] \quad (46)$$

### 4.1.2 Nonlinear MIMO-OFDM system

The analysis presented in this section is based on the assumption that the distortion caused by the HPA can be modeled as additive Gaussian noise, whose variance depends on the input signal and the nonlinearity HPA characteristics. Applying the MMSE equalizer to the received signal $\mathbf{Y}^{\text{NL}}(n,i)$, the following estimate of the transmitted signal is obtained:

$$\hat{\mathbf{X}}(n,i) = [\mathbf{H}^H(n,i)\mathbf{H}(n,i) + \sigma_n^2\mathbf{I}]^{-1}\mathbf{H}^H(n,i) [K_0\mathbf{H}(n,i)\mathbf{X}(n,i) + \mathbf{H}(n,i)\mathbf{D}(n,i) + \mathcal{N}(n,i)] \quad (47)$$

The instantaneous SNR at the receiver in the presence of HPA nonlinearity is

$$\gamma_{\text{NL}} = \frac{K_0^2 \sigma_x^2 |\mathbf{H}(n,i)|^2}{|\mathbf{H}(n,i)|^2 \sigma_d^2 + \sigma_n^2} \quad (48)$$

Let $\gamma_c = |K_0|^2 \sigma_x^2$ and $\mathbf{N} = |\mathbf{H}(n,i)|^2$, and Equation 48 then becomes

$$\gamma_{\text{NL}} = \frac{\gamma_c \mathbf{N}}{\mathbf{N}\sigma_d^2 + \sigma_n^2} = f(\mathbf{N}) = \gamma \quad (49)$$

and the pdf is given by

$$p_{\gamma_{\text{NL}}}(\gamma) = p_{\mathbf{N}}(g(\gamma))g'(\gamma) \quad (50)$$

where $g(\gamma) = \mathbf{N}$ and $g(f(\mathbf{N})) = \mathbf{N}$. Moreover, the instantaneous SNR in the nonlinear case can be expressed by

$$\gamma = \frac{\gamma_c g(\gamma)}{g(\gamma)\sigma_d^2 + \sigma_n^2} \quad (51)$$

From this equation, we can derive $g(\gamma)$ and then determine $g'(\gamma)$, which are given by (52) and (53), respectively:

$$g(\gamma) = \frac{\gamma \sigma_n^2}{\gamma_c - \gamma \sigma_d^2} \quad (52)$$

$$g'(\gamma) = \frac{\gamma_c \sigma_n^2}{(\gamma_c - \gamma \sigma_d^2)^2} \quad (53)$$
Substituting Equations 52 and 53 in Equation 44, we obtain

$$p_{\gamma_{\text{NL}}}(\gamma) = \begin{cases} \frac{\gamma_c \sigma_n^2}{\Omega(\gamma_c - \gamma \sigma_d^2)^2} \exp\left(-\frac{\gamma \sigma_n^2}{\Omega(\gamma_c - \gamma \sigma_d^2)}\right) & 0 \le \gamma \le \frac{\gamma_c}{\sigma_d^2} \\ 0 & \text{otherwise} \end{cases} \quad (54)$$

Then, the average BER of the M-QAM nonlinear MIMO-OFDM system is given by

$$\text{BER}^{\text{NL}} = \int_{0}^{\frac{\gamma_c}{\sigma_d^2}} \frac{4}{\log_2 M} Q\left(\sqrt{\frac{3\gamma_b \log_2 M}{M-1}}\right) \frac{\gamma_c \sigma_n^2}{\Omega(\gamma_c - \gamma \sigma_d^2)^2} \\ \exp\left(-\frac{\gamma \sigma_n^2}{\Omega(\gamma_c - \gamma \sigma_d^2)}\right) d\gamma \quad (55)$$

### 4.1.3 Nonlinear MIMO-OFDM system with RF crosstalk

The detected signal at the receiver level in the presence of HPA nonlinearity and crosstalk can be expressed as

$$\hat{\mathbf{X}}(n, i) = \left[ \begin{aligned} & \mathcal{H}^H(n, i)\mathcal{H}(n, i) + \sigma_n^2 \mathbf{I} \end{aligned} \right]^{-1} \mathcal{H}^H(n, i) \\ & \left[ K_0 \mathcal{H}(n, i)\mathbf{X}(n, i) + K_0 \alpha \mathcal{H}(n, i)\mathbf{\chi}(n, i) \right. \\ & \left. \qquad + \mathcal{H}(n, i)\mathbf{D}(n, i) + \mathcal{N}(n, i) \right] \quad (56)$$

Then, the instantaneous SNR of the received signal is defined as

$$\gamma_{\text{NL+cross}} = \frac{|K_0|^2 \sigma_n^2 |\mathcal{H}(n, i)|^2}{|\mathcal{H}(n, i)|^2 (\sigma_d^2 + |K_0|^2 |\alpha|^2 \sigma_n^2) + \sigma_n^2} \quad (57)$$

Proceeding in the same way as in the previous case, we considerer that $\mathbf{N} = |\mathcal{H}(n, i)|^2, \gamma_c = |K_0|^2 \sigma_n^2$ and $\gamma_e = |K_0|^2 |\alpha|^2 \sigma_n^2$, Equation 57 becomes

$$\gamma_{\text{NL+cross}} = \frac{\gamma_c \mathbf{N}}{\mathbf{N}\sigma_d^2 + \mathbf{N}\gamma_e + \sigma_n^2} = f(\mathbf{N}) = \gamma \quad (58)$$

The instantaneous SNR in the nonlinear with crosstalk can be expressed by

$$\gamma = \frac{\gamma_c g(\gamma)}{g(\gamma)\sigma_d^2 + g(\gamma)\gamma_e + \sigma_n^2} \quad (59)$$

Similar to the nonlinear case, we can deduce $g(\gamma)$ and $g'(\gamma)$:

$$g(\gamma) = \frac{\gamma \sigma_n^2}{\gamma_c - \gamma \sigma_d^2 - \gamma \gamma_e} \quad (60)$$

$$g'(\gamma) = \frac{\gamma_c \sigma_n^2}{(\gamma_c - \gamma \sigma_d^2 - \gamma \gamma_e)^2} \quad (61)$$

and similar to the previous case, we can obtain a closed-form expression for the pdf $p_{\gamma_{\text{NL}}+\text{cross}}$:

$$p_{\gamma_{\text{NL}}+\text{cross}}(\gamma) = \begin{cases} \frac{\gamma_c \sigma_n^2}{\Omega(\gamma_c - \gamma \sigma_d^2 - \gamma \gamma_e)^2} \exp\left(-\frac{\gamma \sigma_n^2}{\Omega(\gamma_c - \gamma \sigma_d^2 - \gamma \gamma_e)}\right) & 0 \le \gamma \le \frac{\gamma_c}{\sigma_d^2 + \gamma_e} \\ 0 & \text{otherwise} \end{cases} \quad (62)$$

Therefore, the BER expression of the M-QAM nonlinear MIMO-OFDM system with crosstalk is given by

$$\text{BER}^{\text{NL+cross}} = \int_{0}^{\frac{\gamma_c}{\sigma_d^2 + \gamma_e}} \frac{4}{\log_2 M} Q\left(\sqrt{\frac{3\gamma_b \log_2 M}{M-1}}\right) \\ \frac{\gamma_c \sigma_n^2}{\Omega(\gamma_c - \gamma \sigma_d^2 - \gamma \gamma_e)^2} \exp\left(-\frac{\gamma \sigma_n^2}{\Omega(\gamma_c - \gamma \sigma_d^2 - \gamma \gamma_e)}\right) d\gamma \quad (63)$$

The integrals presented in Equations 55 and 63 are computed using MATLAB in order to determine the average BER, because the calculation of these integrals is intractable. An approximation for this integral is suggested in full perspective.

## 4.2 System capacity

It is theoretically known that the MIMO system can provide a greater data rate than a conventional wireless system. Thereafter, we focus on the effects of nonlinear amplification due to HPA and nonlinear crosstalk on the system capacity in the MIMO-OFDM system in the presence of an MMSE receiver. The theory expression for system capacity is given by [5]

$$C = \log_2[1 + \gamma] \quad (64)$$

In this analysis, we substitute the SNR expressions determined for different cases (linear (43), nonlinear (48), and nonlinear with crosstalk (57)) in Equation 64. Thus, we obtain the system capacity considered for the three cases considered, which are given by (65), (66), and (67), respectively:

$$C_L = \log_2 \left[ 1 + \frac{\sigma_x^2 |\mathcal{H}(n, i)|^2}{\sigma_n^2} \right] \quad (65)$$

$$C_{\text{NL}} = \log_2 \left[ 1 + \frac{K_0^2 \sigma_x^2 |\mathcal{H}(n, i)|^2}{|\mathcal{H}(n, i)|^2 \sigma_d^2 + \sigma_n^2} \right] \quad (66)$$

$$C_{\text{NL+cross}} = \log_2 \left[ 1 + \frac{|K_0|^2 \sigma_x^2 |\mathcal{H}(n, i)|^2}{|\mathcal{H}(n, i)|^2 (\sigma_d^2 + |K_0|^2 |\alpha|^2 \sigma_x^2) + \sigma_n^2} \right] \quad (67)$$
## 5. Compensation of the joint effects of HPA nonlinearity and crosstalk

In this section, we introduce an iterative estimation and cancellation technique of nonlinearly distorted signals received by the VBLAST MIMO-OFDM transmission system. Indeed, we also propose two compensation methods for two scenarios such as (1) the nonlinear distortion due to HPA without crosstalk and (2) the nonlinear distortion with crosstalk. Accordingly, in scenario 1, we attempt to estimate and mitigate the nonlinear distortion term in Equation 9, and in scenario 2, we try to eliminate jointly the nonlinear distortion due to HPA and the crosstalk between the different branches present in Equation 15.

### 5.1 NDC in nonlinear MIMO-OFDM system

The NDC technique requires knowledge of the HPA parameters. Indeed, the aim of this paper is to estimate and mitigate simultaneously HPA nonlinearity and RF crosstalk basing on the theoretical analysis done in this paper, where we used a polynomial approximation in order to estimate the nonlinear HPA (g[.]), which is described in Section 3.

The proposed method is applied at the receiver side, independently for each antenna using its own HPA parameters, and it consists of the following steps [5,20-22]:

**Step 1:** Using $\hat{X}_k$, an estimate $r_k$ of the original transmitted constellation $X_k$ is obtained by applying hard decoding. This process is carried out for all active carriers. Using the recovered symbols, the time domain signal is reproduced via IFFT as

$$ \hat{\mathbf{x}}_k(n) = \mathbf{V}^H \mathbf{r}_k(n) \qquad (68) $$

**Step 2:** Using the nonlinear model of HPA that is approximated by a polynomial model, which is described

in the previous section, we compute the estimation $\hat{\mathbf{d}}_k^{(q)}(n)$ of the nonlinear distortion terms using Equation 9:

$$ \hat{\mathbf{d}}_k^{(q)}(n) = \hat{g} \left[ \hat{\mathbf{x}}_k^{(q)}(n) \right] - K_0 \hat{\mathbf{x}}_k^{(q)}(n) \qquad (69) $$

where $\hat{\mathbf{x}}_k^{(q)}(n)$ is the time domain representation of the recovered signal at iteration $q$ and $\hat{g}[.]$ is the estimated transfer function.

The frequency domain term is obtained by applying the FFT operator:

$$ \hat{\mathbf{D}}_k^{(q)}(n) = \mathbf{V} \left\{ \hat{g} \left[ \hat{\mathbf{x}}_k^{(q)}(n) \right] - K_0 \hat{\mathbf{x}}_k^{(q)}(n) \right\} \qquad (70) $$

**Step 3:** The distortion $\hat{\mathbf{D}}_k^{(q)}(n)$ is subtracted from the estimated signal $\hat{\mathbf{x}}_k^{(q)}$. Using this result, the transmitted constellation is re-estimated in a new decoding/distortion cancellation step, and the process can be carried out iteratively.

The technique is depicted in Figure 5.

### 5.2 NDC in nonlinear MIMO-OFDM system with crosstalk

We now present the proposed method in order to compensate the joint effects of the nonlinear distortion and crosstalk in the MIMO-OFDM system. Based on the scheme proposed in the previous subsection, we propose the compensation method for the two impairments together. This technique is provided in Figure 6.

Using the same steps above, we can determine the estimation of the nonlinear distortion term $\hat{\mathbf{d}}_k^{(q)}(n)$:

<figure><img src="image_6.png" /><figcaption>Figure 5 Proposed VBLAST MIMO-OFDM receiver structure for iterative NDC of nonlinear distortion using MMSE receiver.</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>Figure 6 Proposed VBLAST MIMO-OFDM receiver structure for iterative NDC of nonlinear crosstalk.</figcaption></figure>

$$ \hat{\mathbf{d}}_{k}^{(q)}(n) = \hat{\mathbf{g}}\left[\hat{\mathbf{x}}_{k}^{(q)}(n) + \sum_{l=1, l \neq k}^{Nt} \alpha_{lk}(n) \otimes \hat{\mathbf{x}}_{l}^{(q)}(n)\right] - K_{0}\left(\hat{\mathbf{x}}_{k}^{(q)}(n) + \sum_{l=1, l \neq k}^{Nt} \alpha_{lk}(n) \otimes \hat{\mathbf{x}}_{l}^{(q)}(n)\right) \quad (71) $$

The frequency domain term is obtained by applying the FFT operator:

$$ \hat{\mathbf{D}}_{k}^{(q)}(n) = V\left\{\hat{\mathbf{g}}\left[\hat{\mathbf{x}}_{k}^{(q)}(n) + \sum_{l=1, l \neq k}^{Nt} \alpha_{lk}(n) \otimes \hat{\mathbf{x}}_{l}^{(q)}(n)\right] - K_{0}\left(\hat{\mathbf{x}}_{k}^{(q)}(n) + \sum_{l=1, l \neq k}^{Nt} \alpha_{lk}(n) \otimes \hat{\mathbf{x}}_{l}^{(q)}(n)\right)\right\} \quad (72) $$

This process can also be performed iteratively.

## 6. Numerical results

In this section, we present theoretical and simulation results illustrating the performance of VBLAST MIMO-OFDM systems with the proposed compensation techniques for HPA nonlinearity and crosstalk, specifically on the average BER and system capacity with 64 subcarriers and 16-QAM. Herein, we consider a Rayleigh flat fading channel that is quasi-stationary on two successive OFDM symbols. The values of the parameters considered for HPA are assumed to be $\alpha_a = 1$, $\beta_a = 1$, $\alpha_p = \frac{\pi}{3}$, and $\beta_p = 1$.

The average BER performances of the nonlinear MIMO-OFDM system and the nonlinear MIMO-OFDM system with crosstalk are shown in Figures 7 and 8, respectively. In this plot, the first set of curves (solid lines) is obtained from simulation results of the system proposed, and the second set of curves (marker lines) is obtained from the theoretical study of the BER using (46), (55), and (63). It can be seen that the results obtained from closed-form expressions are very close to the ones given by the simulation.

Figure 7 illustrates BER versus $Eb/N_0$ for IBO = {4-8}. As illustrated, $Eb/N_0$ imposes a great impact on the BER performance. When IBO is decreasing (e.g., 4 dB), the modulated input signal can be higher than $A_0$ with a strong probability. This gives an irreducible error at higher $Eb/N_0$.

To further illustrate the effect of the crosstalk in the nonlinear MIMO-OFDM system, Figure 8 shows the average BER for different values of the crosstalk parameters, fixing $IBO = 8$ dB. The residual degradation on the performance of the average BER, due to the nonlinear crosstalk, increases as the parameters ($\alpha$) become larger.

The system capacity (using Equations 65, 66, and 67) for the nonlinear MIMO-OFDM system and the nonlinear MIMO-OFDM system with crosstalk is plotted in Figures 9 and 10, respectively.
<figure><img src="image_8.png" /><figcaption>**Figure 7 Theoretical and simulation results for BER vs SNR for MIMO-OFDM systems.** Polynomial approximation of HPA with 16-QAM, $N_c = 64$, $N_t = N_r = 2$, and $IBO = \{4-8-10\}$ dB.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**Figure 8 Theoretical and simulation results for BER vs SNR for MIMO-OFDM systems.** Polynomial approximation of HPA with 16-QAM, $N_c = 64$, $N_t = N_r = 2$, $IBO = 8$ dB, and different values of crosstalk.</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>Figure 9 Capacity evaluation in MIMO-OFDM system with linear and nonlinear HPA for different values of IBO. Nc = 64 and Nt = Nr = 2.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 10 Capacity evaluation in MIMO-OFDM system with linear and nonlinear HPA for different values of crosstalk. Nc = 64, IBO = 8 dB, and Nt = Nr = 2.</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>Figure 11 BER versus Eb/N0 of MIMO-OFDM system with NDC technique for Nc = 64, Nt = Nr = 2, and IBO = 8 dB.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 12 BER versus Eb/N0 of MIMO-OFDM system with NDC technique for Nc = 64, Nt = Nr = 2, IBO = 8 dB, and crosstalk = -8 dB.</figcaption></figure>
Figure 9 show the effect of the nonlinearities due to HPA on the system capacity with different values of IBO. As noticed, the nonlinear distortion result causes a decrease in the capacity of MIMO-OFDM systems. On the other hand, the capacity of the system improves with the increase in the value of the IBO.

In Figure 10, we show the effect of the crosstalk on the system capacity as a function of Eb/N0, taking into account the HPA nonlinearity, for IBO = 8 dB. As observed, the system capacity increases as the crosstalk parameters (α) become larger. However, profiting of the RF crosstalk can improve the system capacity.

Finally, Figures 11 and 12 show that the nonlinear distortion and the nonlinear distortion with crosstalk in MIMO-OFDM systems can be effectively compensated using the NDC technique even after one iteration. Moreover, an increase in the number of iterations gives a significant improvement in the estimation procedure, although it slightly increases the complexity of implementation.

## 7. Conclusions

In this paper, we have investigated and analyzed the joint effects of nonlinear HPA and crosstalk on the performance of the M-QAM VBLAST MIMO-OFDM system, in terms of BER and system capacity over a Rayleigh fading channel. In addition, new closed-form pdf signal-to-noise ratios are derived in order to develop theoretical expressions of the average BER in different scenarios: nonlinear MIMO-OFDM system and nonlinear MIMO-OFDM system with crosstalk.

On the other hand, we have introduced a compensation method (NDC) that aims to estimate and mitigate simultaneously the HPA nonlinearity and RF crosstalk at the receiver side. This proposed method is based on the theoretical analysis done in this work.

Numerical results showing the joint effects of HPA nonlinearity and crosstalk on the performances of MIMO-OFDM were evaluated and compared. In fact, the results obtained using the theoretical expressions of BER are very close to the simulation results, and it was shown that the NDC method can efficiently compensate these two impairments even after one iteration.

## Competing interests

The authors declare that they have no competing interests.

Received: 29 June 2013 Accepted: 13 March 2014

Published: 22 April 2014

## References

1. SA Haider, K Noori, System design and performance analysis of layered MIMO OFDM communication systems, in *ICTA 2009* (Phoenix Park, 15–18 Feb 2009)

2. SA Bassam, M Helaoui, FM Ghannouchi, Crossover digital predistorter for the compensation of crosstalk and nonlinearity in MIMO transmitters. IEEE Trans. Microw. Theor. Tech. 57(5), 1119–1128 (2009)

3. D Dardari, V Tralli, A Vacarri, A theoretical characterization of nonlinear distortion effects in OFDM systems. IEEE Trans. Comm. 48(10), 1755–1764 (2000)

4. J Qi, S Aissa, On the power amplifier nonlinearity in MIMO transmit beamforming systems. IEEE Trans. Comm. 60(3), 876–887 (2012)

5. F Gregorio, S Werner, TI Laakso, J Cousseau, Receiver cancellation technique for nonlinear power amplifier distortion in SDMA-OFDM systems. IEEE Trans. Veh. Tech. 56(5), 2499–2516 (2007)

6. R Zayani, R Bouallegue, D Roviras, Crossover neural network predistorter for the compensation of crosstalk and nonlinearity in MIMO OFDM systems, in *PIMRC 2010* (Istanbul, 26–30 Sept 2010)

7. F Gregorio, J Cousseau, S Werner, T Riihonen, R Wichman, Power amplifier linearization technique with IQ imbalance and crosstalk compensation for broadband MIMO-OFDM transmitters. Eurasip Journal on Advances in Signal Processing (2011). doi: 10.1186/1687-6180-2011-19

8. AAM Saleh, Frequency-independent and frequency-dependent nonlinear models of TWT amplifiers. IEEE Trans. Comm. 29, 1715–1720 (1981)

9. R Zayani, R Bouallegue, D Roviras, Adaptative predistortions based on neural networks associated with Levenberg-Marquardt algorithm for satellite down links. EURASIP JWCN (2008). doi:10.1155/2008/132729

10. F H Gregorio, S Werner, JE Cousseau, TI Laakso, Iterative channel estimation for multiuser OFDM systems in the presence of power amplifier nonlinearities, in *PIMRC'06* (Helsinki, 11–14 Sept 2006), p. 11

11. H Bouhadda, R Zayani, R Bouallegue, D Roviras, Levenberg memory crossover neural network predistorter for the compensation of memory crosstalk and HPA, in *19th European Signal Processing Conference EUSIPCO* (Barcelona, 29 Aug–2 Sept 2011)

12. W-C Hua, P-T Lin, C-P Lin, C-Y Lin, H-L Chang, CW Liu, T-Y Yang, G-K Ma, Coupling effects of dual SiGe power amplifiers for 802.11n MIMO applications, in *IEEE Radio Freq Integr Circuits Symp* (San Francisco, 11–13 June 2006), p. 4

13. P-J Bouvet, V Le Nir, M H'elard, R Le Gouable, Spatial multiplexed coded MC-CDMA with iterative receiver, in *Proceedings of the IEEE 15th International Symposium on Personal, Indoor, and Mobile Radio Communications* (Barcelona, 8 Sept 2004), pp. 5–8

14. P Liu, MQ Wu, CX Xu, F Zheng, Multi-user MIMO linear precoding schemes in OFDM systems, in *IEEE 3rd ICCSIT* (Chengdu, 9–11 July 2010)

15. M Dakhli, R Zayani, R Bouallegue, Compensation for nonlinear distortion in MIMO OFDM systems using MMSE receiver, in *IEEE 15th International Conference on Communication Systems, ICCS* (Singapore, 21–23 Nov 2012)

16. A Conti, D Dardari, V Tralli, An analytical framework for CDMA systems with a nonlinear amplifier and AWGN. IEEE Trans. Comm. 50(7), 1110–1120 (2002)

17. R Raich, H Qian, GT Zhou, Orthogonal polynomials for power amplifier modeling and predistorter design. IEEE Trans. Veh. Tech. 53(5), 1468–1479 (2004)

18. V Meghdadi, BER calculation, in *Wireless Communications*, ed. by A Goldsmith, 2008

19. M Agrawal, Y Raut, BER analysis of MIMO OFDM system for AWGN & Rayleigh fading channel. Int J Comput Appl 34, 9 (2011)

20. P Drotar, J Gazda, M Deumal, P Galajda, D Kocur, Receiver based compensation of nonlinear distortion in MIMO-OFDM, in *IEEE IMWS* (Aveiro, 22–23 Feb 2010)

21. P Drotar, J Gazda, P Galajda, D Kocur, P Pavelka, Receiver technique for iterative estimation and cancellation of nonlinear distortion in MIMO SFBC-OFDM (IEEE Trans, Consumer Electronics, 2010)

22. M Dakhli, R Zayani, R Bouallegue, A theoretical characterization and compensation of nonlinear distortion effects and performance analysis using polynomial model in MIMO OFDM systems under Rayleigh fading channel, in *18th IEEE Symposium on Computers and Communications, ISCC* (Split, 7–10 July 2013)

doi:10.1186/1687-1499-2014-61

**Cite this article as:** Dakhli et al.: Theoretical analysis and compensation for the joint effects of HPA nonlinearity and RF crosstalk in VBLAST MIMO-OFDM systems over Rayleigh fading channel. EURASIP Journal on Wireless Communications and Networking 2014 **2014**:61.