## 1 Introduction  
In digital communication, strong nonlinear effects throughout the transmission chain can have a negative impact on the overall system's performance [1]. Pre-processing associated with massive MIMO systems can lead to signals with envelope fluctuations, which are very prone to nonlinear effects, especially when massive MIMO schemes are combined with OFDM modulations [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. Several works focused on MIMO systems and their important parameters such as mutual coupling [11, 12, 13, 14]. Pre-processing techniques employed at the transmitter side of MIMO systems, such as the ones used by MIMO singular value decomposition (MIMO-SVD) techniques, can lead to signals with large envelope fluctuations and high peak to-average power ratio (PAPR) [3]. The same occurs with the Zero Forcing Transmitter (ZFT), Maximum Ratio Transmitter (MRT), and Equal Gain Transmitter (EGT), when used in m-MIMO implementation with pre- or post-processing [4]. On the other hand, several authors have qualified nonorthogonal multiple access (NOMA) as the main candidate to support fifth-generation (5G) wireless communication. This has been investigated in several works (see, e.g., [15, 16, 17]). By combining signals with different power levels, NOMA schemes can also be very prone to nonlinear effects. If the nonlinear characteristics that affect a given system are known, we can design appropriate receivers that take into account the nonlinear effects introduced by the transmitter. In fact, efficient receivers that jointly perform detection and estimation and cancellation of nonlinear distortion effects, can have excellent performance/complexity trade-offs [2; 8]. On the other hand, taking into account nonlinear distortion effects in the receiver design can have significant performance improvements [18], sometimes even outperforming the linear case [10; 18; 19; 20]. As an alternative, we can employ receivers such as the Bussgang receiver, which estimate and cancel nonlinear distortion [21; 22; 23]. All these techniques require the knowledge of the nonlinear characteristic used by the transmitter, which can be obtained blindly using higher-order cumulants (HOC) theory. We propose to study the issue of nonlinear system identification using the fifth-order cumulants domain. One of the first uses of HOC in signal processing has been devoted to the problem of blind system identification. These systems, which can be linear or nonlinear, are excited by an unobservable, zero-mean, independent and identically distributed (i.i.d), stationary, ergodic and non-Gaussian signal under the presence of Gaussian noise. Linear systems identification has been very deeply investigated in the literature (see, e.g., [24; 25; 26; 27; 28]). The primary existing blind approaches are based on second-order cumulants (usually called autocorrelation function). The second- and third-order cumulants were used by [PERSON] et al. to identify the nonminimum phase (NMP) of finite impulse response (FIR) systems [24]. Likewise, [PERSON] et al. [25] presented an identification method by using both second- and higher-order statistics of an NMP output system and many approaches devoted to the same issue have been reported in the literature (see, e.g., [26]). However, these second-order-based methods are not able to identify correctly the linear system's model and their performance was degraded when the output is affected by additive Gaussian noise. To overcome these inconveniences, other linear algebraic methods involving only HOC of output system have been reported in the literature (see, e.g., [29; 30; 31; 32; 33; 34]), which present several advantages such as effectiveness to reject additive Gaussian noise and ability to reconstruct the phase. Other authors showed, both analytically and by simulations, that better results can be obtained using these approaches. The case when the noise is non-Gaussian was studied deeply by [PERSON] et al. by using HOC tools [35].