# Machine learning based automatic modulation recognition for wireless communications: a comprehensive survey

Bachir Jdid, Kais Hassan, Iyad Dayoub, Wei Hong Lim, Mastaneh Mokayef

► To cite this version:

Bachir Jdid, Kais Hassan, Iyad Dayoub, Wei Hong Lim, Mastaneh Mokayef. Machine learning based automatic modulation recognition for wireless communications: a comprehensive survey. IEEE Access, 2021, 9, pp.57851-57873. 10.1109/ACCESS.2021.3071801. hal-03264908

HAL Id: hal-03264908

https://hal.science/hal-03264908

Submitted on 2 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Received March 17, 2021, accepted April 5, 2021, date of publication April 8, 2021, date of current version April 21, 2021.

Digital Object Identifier 10.1109/ACCESS.2021.3071801

# Machine Learning Based Automatic Modulation Recognition for Wireless Communications: A Comprehensive Survey

BACHIR JIDID<sup>1</sup>, KAIS HASSAN<sup>1,2</sup>, IYAD DAYOUB<sup>1,3,4</sup>, (Senior Member, IEEE), WEI HONG LIM<sup>1</sup>, (Senior Member, IEEE), AND MASTANEH MOKAYEF<sup>1</sup>

Corresponding author: Iyad Dayoub (iyad.dayoub@uphf.fr)

This work was supported by the Ministry of Higher Education Malaysia through the Fundamental Research Grant Scheme under Grant Proj-FRGS/1/2019/TK04/UCSI/02/1 and Grant Proj-FRGS/1/2020/TK0/UCSI/02/4.

**ABSTRACT** The rapid development of information and wireless communication technologies together with the large increase in the number of end-users have made the radio spectrum more crowded than ever. Besides, providing a stable and reliable service is challenging, as electromagnetic environments are evolving and becoming more sophisticated. Accordingly, there is an urgent need for more reliable and intelligent communication systems that can improve the spectrum efficiency and the quality of service to provide agile management of network resources, so as to better meet the needs of future wireless users. Specifically, Automatic Modulation Recognition (AMR) plays an essential role in most intelligent communication systems especially with the emergence of Software Defined Radio (SDR). AMR is an indispensable task while performing spectrum sensing in Cognitive Radio (CR). Thanks to the significant advancements in Deep Learning (DL) applications, new and powerful tools have been provided which can tackle problems in this space. Thus, today, integrating DL models into AMR has gained the attention of many researchers. This work aims to provide a comprehensive state-of-the-art review of the most recent Machine Learning (ML) based AMR methods for Single-Input Single-Output (SISO) and Multiple-Input Multiple-Output (MIMO) systems. Furthermore, the architecture of each model will be identified along with a detailed comparison in terms of specifications and performance. Finally, an outline of the open problems, challenges, and potential research directions is provided along with discussion and conclusion.

**INDEX TERMS** Automatic modulation recognition, deep learning, machine learning, MIMO, SISO, wireless signal classification.

## I. INTRODUCTION

AMR was initially motivated by its significance in many military applications such as electronic warfare, intelligence, surveillance, and threat analysis. It is a crucial task in intercepting the communications between adversary units and recovering the intercepted signal [1]. Moreover, from the standpoint of communication security, modulations can serve as another level of encryption. Specifically, an encryption

The associate editor coordinating the review of this manuscript and approving it for publication was Stefan Schwarz.

key may be integrated into the received signal so that the AMR process can verify the legitimacy of the received signal to avoid emulation attacks and jam.

Today, the new generation of communication standards (5G & beyond) integrates multiple technologies of communication and information processing. This integration increases the cognition of intelligent communication systems and allows operators to reduce operational and maintenance costs plus attracts more users. An intelligent communication system is a system that can generate and implement new services quickly, easily, flexibly, economically, and efficiently [2].

<footnote n="1"><sup>1</sup>Faculty of Engineering Technology and Built Environment, UCSI University, Kuala Lumpur 56000, Malaysia</footnote>

<footnote n="2"><sup>2</sup>Laboratoire d'Acoustique de l'Université du Mans (LAUM) -UMR CNRS 6613, Le Mans University, 72085 Le Mans, France</footnote>

<footnote n="3"><sup>3</sup>CNRS, Université Polytechnique Hauts-de-France, Université de Lille, ISEN, Centrale Lille, UMR 8520, Département d'Opto-Acousto-Électronique (DOAE), Institut d'Électronique de Microélectronique et de Nanotechnologie (IEMN), 59313 Valenciennes, France</footnote>

<footnote n="4"><sup>4</sup>INSA des Hauts de France, 01100 Bellignat, France</footnote>
Received March 17, 2021, accepted April 5, 2021, date of publication April 8, 2021, date of current version April 21, 2021.

Digital Object Identifier 10.1109/ACCESS.2021.3071801

# Machine Learning Based Automatic Modulation Recognition for Wireless Communications: A Comprehensive Survey

BACHIR JIDID<sup>1</sup>, KAIS HASSAN<sup>1,2</sup>, IYAD DAYOUB<sup>1,3,4</sup>, (Senior Member, IEEE), WEI HONG LIM<sup>1</sup>, (Senior Member, IEEE), AND MASTANEH MOKAYEF<sup>1</sup>

Corresponding author: Iyad Dayoub (iyad.dayoub@uphf.fr)

This work was supported by the Ministry of Higher Education Malaysia through the Fundamental Research Grant Scheme under Grant Proj-FRGS/1/2019/TK04/UCSI/02/1 and Grant Proj-FRGS/1/2020/TK0/UCSI/02/4.

**ABSTRACT** The rapid development of information and wireless communication technologies together with the large increase in the number of end-users have made the radio spectrum more crowded than ever. Besides, providing a stable and reliable service is challenging, as electromagnetic environments are evolving and becoming more sophisticated. Accordingly, there is an urgent need for more reliable and intelligent communication systems that can improve the spectrum efficiency and the quality of service to provide agile management of network resources, so as to better meet the needs of future wireless users. Specifically, Automatic Modulation Recognition (AMR) plays an essential role in most intelligent communication systems especially with the emergence of Software Defined Radio (SDR). AMR is an indispensable task while performing spectrum sensing in Cognitive Radio (CR). Thanks to the significant advancements in Deep Learning (DL) applications, new and powerful tools have been provided which can tackle problems in this space. Thus, today, integrating DL models into AMR has gained the attention of many researchers. This work aims to provide a comprehensive state-of-the-art review of the most recent Machine Learning (ML) based AMR methods for Single-Input Single-Output (SISO) and Multiple-Input Multiple-Output (MIMO) systems. Furthermore, the architecture of each model will be identified along with a detailed comparison in terms of specifications and performance. Finally, an outline of the open problems, challenges, and potential research directions is provided along with discussion and conclusion.

**INDEX TERMS** Automatic modulation recognition, deep learning, machine learning, MIMO, SISO, wireless signal classification.

## I. INTRODUCTION

AMR was initially motivated by its significance in many military applications such as electronic warfare, intelligence, surveillance, and threat analysis. It is a crucial task in intercepting the communications between adversary units and recovering the intercepted signal [1]. Moreover, from the standpoint of communication security, modulations can serve as another level of encryption. Specifically, an encryption

The associate editor coordinating the review of this manuscript and approving it for publication was Stefan Schwarz.

key may be integrated into the received signal so that the AMR process can verify the legitimacy of the received signal to avoid emulation attacks and jam.

Today, the new generation of communication standards (5G & beyond) integrates multiple technologies of communication and information processing. This integration increases the cognition of intelligent communication systems and allows operators to reduce operational and maintenance costs plus attracts more users. An intelligent communication system is a system that can generate and implement new services quickly, easily, flexibly, economically, and efficiently [2].

<footnote n="1"><sup>1</sup>Faculty of Engineering Technology and Built Environment, UCSI University, Kuala Lumpur 56000, Malaysia</footnote>

<footnote n="2"><sup>2</sup>Laboratoire d'Acoustique de l'Université du Mans (LAUM) -UMR CNRS 6613, Le Mans University, 72085 Le Mans, France</footnote>

<footnote n="3"><sup>3</sup>CNRS, Université Polytechnique Hauts-de-France, Université de Lille, ISEN, Centrale Lille, UMR 8520, Département d'Opto-Acousto-Électronique (DOAE), Institut d'Électronique de Microélectronique et de Nanotechnologie (IEMN), 59313 Valenciennes, France</footnote>

<footnote n="4"><sup>4</sup>INSA des Hauts de France, 01100 Bellignat, France</footnote>
It is recognized by its autonomy in making decisions based on its external conditions; and its ability of transmission environment cognition. In this direction, AMR can improve the awareness of such systems and provides interference detection and spectrum management. For instance, AMR is considered as a major task in CR which can provide dynamic radio resource management by employing re-configurable software defined transceivers. These transceivers can reconfigure their transmission parameters based on the available communication resources in the electromagnetic environment [3]. However, it is worth mentioning that the term 'automatic' is the opposite of the initial implementation of manual modulation recognition.

In modern communication systems, a pool of modulation types can be used by the transmitter to control both the data rate and the bandwidth usage. While the transmitter selects the modulation type adaptively, the receiving end may or may not know the modulation type. Correspondingly, the modulation information can be included in each signal frame so that the receiver would have the knowledge of the modulation type and react accordingly. However, the frequency spectrum is extremely limited and, thus, this strategy may not be efficient enough in real scenarios since it will affect the spectrum efficiency due to the extra information in each signal frame [4]–[7]. In fact, today’s wireless networks are highly heterogeneous and the number of users is increasing significantly. Hence, AMR mechanism can be used to detect the modulation type of the received signals and, thus, eliminate any potential overhead in the network protocol. Ultimately, the signals will be correctly demodulated and the received data will be accurately recovered. Nowadays, the receivers enjoy a high computational power, especially with the advancements in microprocessors. Thus, the signal processing required in AMR becomes more feasible [4].

In literature, most of the existing AMR algorithms are typically implemented via two main approaches: Likelihood-Based (LB) and Feature-based (FB). On one hand, LB approaches calculate the likelihood functions of all candidate modulation schemes of the received signal and select the scheme with maximal likelihood value [3]. Several LB-based AMR techniques have been introduced for MIMO systems to blindly and semi-blindly identify the modulation scheme especially when the channel suffers from severe spatial correlation. Theoretically, LB approaches can provide the optimal solution but they suffer from an extremely high computational complexity and lack of robustness against the model mismatch. These undesirable drawbacks tend to restrict the feasibility of LB-AMR to be implemented in real-time and low-cost applications. On the other hand, FB approaches are considered to be a good alternative for LB approaches because they can produce suboptimal solutions with much lower computational complexity.

Nowadays, ML and DL applications have shown overwhelming advantages in wide-ranging fields such as computer vision, healthcare, robotics, and communication systems. Typically, ML algorithms need some guidance from

expert engineers to make the required adjustments when these algorithms fail in the prediction process. On the contrary, DL networks can learn unsupervised from unlabelled or unstructured data, and make decisions without any human supervision. In fact, DL imitates the functionality of the human brain in processing data and creating patterns to be used in decisions making. However, integrating ML and DL in wireless communication networks and systems has great potential for their development since it can make them more intelligent. This suggests that integrating ML and DL approaches in AMR process can achieve significant results and boost its performance.

## A. RELATED SURVEYS

In the literature, there are some brief surveys on DL-AMR<fnref n="1" /> for SISO communication systems, e.g., [8] and [9]. For instance, the third section in [8] was dedicated to giving a brief overview of modulation recognition approaches based on the category of the extracted features. The authors focused mainly on briefing the AMR methods based on crafted expert features and developed ML classifiers. However, several DL-AMR models were also reviewed in terms of the employed features and classification criteria. Likewise, in [9], a brief review of DL-AMR techniques in terms of modulation pool are provided. The main DL models in the AMR methods were outlined along with a demonstration on the feasibility of using Convolutional Neural Network (CNN) to recognize wireless signals. Table 1 provides a comparison between the existing surveys and our paper.

## B. SCOPE AND OBJECTIVE OF THE SURVEY

This survey aims to provide a comprehensive view on state-of-the-art ML and DL practices in AMR domain for SISO and MIMO wireless communication systems. In this survey, a detailed and fair comparison between the latest methods in this area is provided along with emphasizing their limitations and key advantages. Each method is presented in a clear and concise manner along with all of its aspects. For instance, the features extraction and classification criteria along with channel assumption and modulation sets are pointed out for ML-AMR methods. Furthermore, DL-AMR methods are compared in terms of network architecture, network training and testing parameters, used datasets, classification accuracy, and concluding remarks.

However, there are several architectural parameters that must be considered when designing a DL model. These parameters have critical impacts on designing or choosing the right DL model for solving the required AMR problem. For instance, whether the input data has a fixed-length or varying one. Besides, the depth of the network is a key parameter since it may simplify or complicate the task of modulation scheme classification. Equally, using an insufficient or badly constructed dataset can distort the hyperplanes of classifiers

<footnote n="1">¹For simplicity, we refer to ML-based AMR and DL-based AMR by ML-AMR and DL-AMR, respectively.</footnote>
It is recognized by its autonomy in making decisions based on its external conditions; and its ability of transmission environment cognition. In this direction, AMR can improve the awareness of such systems and provides interference detection and spectrum management. For instance, AMR is considered as a major task in CR which can provide dynamic radio resource management by employing re-configurable software defined transceivers. These transceivers can reconfigure their transmission parameters based on the available communication resources in the electromagnetic environment [3]. However, it is worth mentioning that the term 'automatic' is the opposite of the initial implementation of manual modulation recognition.

In modern communication systems, a pool of modulation types can be used by the transmitter to control both the data rate and the bandwidth usage. While the transmitter selects the modulation type adaptively, the receiving end may or may not know the modulation type. Correspondingly, the modulation information can be included in each signal frame so that the receiver would have the knowledge of the modulation type and react accordingly. However, the frequency spectrum is extremely limited and, thus, this strategy may not be efficient enough in real scenarios since it will affect the spectrum efficiency due to the extra information in each signal frame [4]–[7]. In fact, today’s wireless networks are highly heterogeneous and the number of users is increasing significantly. Hence, AMR mechanism can be used to detect the modulation type of the received signals and, thus, eliminate any potential overhead in the network protocol. Ultimately, the signals will be correctly demodulated and the received data will be accurately recovered. Nowadays, the receivers enjoy a high computational power, especially with the advancements in microprocessors. Thus, the signal processing required in AMR becomes more feasible [4].

In literature, most of the existing AMR algorithms are typically implemented via two main approaches: Likelihood-Based (LB) and Feature-based (FB). On one hand, LB approaches calculate the likelihood functions of all candidate modulation schemes of the received signal and select the scheme with maximal likelihood value [3]. Several LB-based AMR techniques have been introduced for MIMO systems to blindly and semi-blindly identify the modulation scheme especially when the channel suffers from severe spatial correlation. Theoretically, LB approaches can provide the optimal solution but they suffer from an extremely high computational complexity and lack of robustness against the model mismatch. These undesirable drawbacks tend to restrict the feasibility of LB-AMR to be implemented in real-time and low-cost applications. On the other hand, FB approaches are considered to be a good alternative for LB approaches because they can produce suboptimal solutions with much lower computational complexity.

Nowadays, ML and DL applications have shown overwhelming advantages in wide-ranging fields such as computer vision, healthcare, robotics, and communication systems. Typically, ML algorithms need some guidance from

expert engineers to make the required adjustments when these algorithms fail in the prediction process. On the contrary, DL networks can learn unsupervised from unlabelled or unstructured data, and make decisions without any human supervision. In fact, DL imitates the functionality of the human brain in processing data and creating patterns to be used in decisions making. However, integrating ML and DL in wireless communication networks and systems has great potential for their development since it can make them more intelligent. This suggests that integrating ML and DL approaches in AMR process can achieve significant results and boost its performance.

## A. RELATED SURVEYS

In the literature, there are some brief surveys on DL-AMR<fnref n="1" /> for SISO communication systems, e.g., [8] and [9]. For instance, the third section in [8] was dedicated to giving a brief overview of modulation recognition approaches based on the category of the extracted features. The authors focused mainly on briefing the AMR methods based on crafted expert features and developed ML classifiers. However, several DL-AMR models were also reviewed in terms of the employed features and classification criteria. Likewise, in [9], a brief review of DL-AMR techniques in terms of modulation pool are provided. The main DL models in the AMR methods were outlined along with a demonstration on the feasibility of using Convolutional Neural Network (CNN) to recognize wireless signals. Table 1 provides a comparison between the existing surveys and our paper.

## B. SCOPE AND OBJECTIVE OF THE SURVEY

This survey aims to provide a comprehensive view on state-of-the-art ML and DL practices in AMR domain for SISO and MIMO wireless communication systems. In this survey, a detailed and fair comparison between the latest methods in this area is provided along with emphasizing their limitations and key advantages. Each method is presented in a clear and concise manner along with all of its aspects. For instance, the features extraction and classification criteria along with channel assumption and modulation sets are pointed out for ML-AMR methods. Furthermore, DL-AMR methods are compared in terms of network architecture, network training and testing parameters, used datasets, classification accuracy, and concluding remarks.

However, there are several architectural parameters that must be considered when designing a DL model. These parameters have critical impacts on designing or choosing the right DL model for solving the required AMR problem. For instance, whether the input data has a fixed-length or varying one. Besides, the depth of the network is a key parameter since it may simplify or complicate the task of modulation scheme classification. Equally, using an insufficient or badly constructed dataset can distort the hyperplanes of classifiers

<footnote n="1">¹For simplicity, we refer to ML-based AMR and DL-based AMR by ML-AMR and DL-AMR, respectively.</footnote>
<table><caption>**TABLE 1.** Summary and comparison of existing surveys related to ML-AMR and DL-AMR. The symbol ✓ indicates that a survey includes publications in the scope of a domain. The numbers indicate the count of reviewed publications in the scope of a domain. The symbol ✓ indicates that a survey compares publications in terms of this aspect. The symbol indicates that a survey did not review the scope of a domain.</caption><thead><tr><th>Publication</th><th>One-sentence summary</th><th>ML-AMR</th><th>DL-AMR</th><th>SISO systems</th><th>MIMO systems</th><th>DL network structure</th><th>Training & testing configurations</th><th>Performance & accuracy</th><th>Input formats</th><th>Modulations pool</th></tr></thead><tbody><tr><td>Zhou et al. [9]</td><td>A brief review of the most widely used DL techniques for recognizing a wireless signal in terms of modulation schemes</td><td>-</td><td>23</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Li et al. [8] (the 3rd section)</td><td>A brief overview of emerging ML and DL approaches for signal recognition</td><td>10</td><td>25</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>This survey [8]</td><td>A comprehensive state-of-the-art review of the existing ML-AMR and DL-AMR methods for SISO and MIMO systems</td><td>19</td><td>75</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr></tbody></table>

and, hence, affect badly its accuracy. Trainable parameters are very essential since they reflect the computational complexity and the required storage of the model and, thus, reveal the possibilities of implementing the model on real hardware [10]. Similarly, there are many other parameters to be considered during the model training process such as the batch normalization, activation functions, cost functions, and training configurations [10]. The training configurations include the number of batches, number of epochs, dropout, optimization and regularization algorithms applied during the training process. Ultimately, all of these parameters can affect the accuracy of the DL-based classifier and, thus, should be pointed out. To the best of our knowledge, a comprehensive survey of ML-AMR and DL-AMR methods considering these parameters does not exist in the literature. Moreover, few works are dedicated to reviewing the existing methods for both SISO and MIMO systems.

This article fills this gap by carrying a comprehensive up-to-date survey of researches in these domains. Beyond reviewing the most relevant literature, we discuss the feasibility of various DL architectures in view of solving AMR issues. At the end of this paper, we provide potential future research directions along with open challenges that need more investigation. Hence, our ultimate objective is to provide future affiliated researchers, who plan to take advantage of DL models to resolve problems in the field of AMR, with a definite guide that answers the following key questions:

* Which are the most recent ML and DL models in the field of AMR for MIMO and SISO systems?

* Why are ML and DL promising in AMR domain?

* What should be considered when designing a DL model for AMR?

* What are the most important and promising directions worthy of further investigation?

### C. CONTRIBUTIONS OF THE SURVEY

The aforementioned surveys provide partial answers to some of these questions. This survey goes beyond the previous works and covers a wide range of DL models that have not been explicitly discussed in earlier surveys, e.g., [9]. Unlike such existing surveys, we also review the network architecture, accuracy against Signal-to-Noise Ratio (SNR)

range, training parameters, and key concepts. We also review DL-AMR methods not looked at in other related surveys, including those for MIMO communication systems. While our main scope remains the DL models in AMR domain, for completeness we also discuss the most relevant ML models connected to AMR field. It is worth mentioning that we differentiate between DL-AMR, which refers to the models where the features are automatically extracted from data that has a complex structure and inner correlations, and ML-AMR, where features hand-crafting is required to be done by algorithms designed by expert engineers. Hence, the main key perspectives which distinguish this paper from earlier surveys can be outlined as follows:

* We provide an inclusive review of the existing DL models in AMR domain for both SISO and MIMO systems, instead of briefly discussing the main DL models or focusing on a single type of communication system, e.g. SISO systems [8], [9].

* We particularly discuss the most recent DL models from the perspective of AMR, focusing on their applicability to this area and their structures and parameters.

To the best of our knowledge, this is the first time that AMR is inclusively reviewed for both SISO and MIMO systems in wireless communications from a DL angle.

### D. ORGANIZATION OF THE SURVEY

This paper is structured in a top-down manner as shown in Fig. 1. The outline of this paper can be summarised as:

* First, we begin, in section II, by presenting a basic background of the traditional process of FB-AMR followed by providing a basic ML and DL background to illustrate the main required concepts for the rest of this article. Thereafter, we discuss the main key factors that encourage the implementation of ML and DL models in AMR domain, aiming to clarify why such models can outperform the traditional AMR methods.

* Later in section III, a discussion on the integration of ML models into AMR methods is provided for SISO and MIMO systems.

* Since DL models require large-scale datasets, we begin section IV by presenting the specifications of some existing radio signals datasets. This provides the required information about the signal parameters and
<table><caption>TABLE 1. Summary and comparison of existing surveys related to ML-AMR and DL-AMR. The symbol ✓ indicates that a survey includes publications in the scope of a domain. The numbers indicate the count of reviewed publications in the scope of a domain. The symbol ✓ indicates that a survey compares publications in terms of this aspect. The symbol indicates that a survey did not review the scope of a domain.</caption><thead><tr><th>Publication</th><th>One-sentence summary</th><th>ML-AMR</th><th>DL-AMR</th><th>SISO systems</th><th>MIMO systems</th><th>DL network structure</th><th>Training & testing configurations</th><th>Performance & accuracy</th><th>Input formats</th><th>Modulations pool</th></tr></thead><tbody><tr><td>Zhou et al. [9]</td><td>A brief review of the most widely used DL techniques for recognizing a wireless signal in terms of modulation schemes</td><td>-</td><td>23</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Li et al. [8] (the 3rd section)</td><td>A brief overview of emerging ML and DL approaches for signal recognition</td><td>10</td><td>25</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>This survey [8]</td><td>A comprehensive state-of-the-art review of the existing ML-AMR and DL-AMR methods for SISO and MIMO systems</td><td>19</td><td>75</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr></tbody></table>

and, hence, affect badly its accuracy. Trainable parameters are very essential since they reflect the computational complexity and the required storage of the model and, thus, reveal the possibilities of implementing the model on real hardware [10]. Similarly, there are many other parameters to be considered during the model training process such as the batch normalization, activation functions, cost functions, and training configurations [10]. The training configurations include the number of batches, number of epochs, dropout, optimization and regularization algorithms applied during the training process. Ultimately, all of these parameters can affect the accuracy of the DL-based classifier and, thus, should be pointed out. To the best of our knowledge, a comprehensive survey of ML-AMR and DL-AMR methods considering these parameters does not exist in the literature. Moreover, few works are dedicated to reviewing the existing methods for both SISO and MIMO systems.

This article fills this gap by carrying a comprehensive up-to-date survey of researches in these domains. Beyond reviewing the most relevant literature, we discuss the feasibility of various DL architectures in view of solving AMR issues. At the end of this paper, we provide potential future research directions along with open challenges that need more investigation. Hence, our ultimate objective is to provide future affiliated researchers, who plan to take advantage of DL models to resolve problems in the field of AMR, with a definite guide that answers the following key questions:

* Which are the most recent ML and DL models in the field of AMR for MIMO and SISO systems?

* Why are ML and DL promising in AMR domain?

* What should be considered when designing a DL model for AMR?

* What are the most important and promising directions worthy of further investigation?

### C. CONTRIBUTIONS OF THE SURVEY

The aforementioned surveys provide partial answers to some of these questions. This survey goes beyond the previous works and covers a wide range of DL models that have not been explicitly discussed in earlier surveys, e.g., [9]. Unlike such existing surveys, we also review the network architecture, accuracy against Signal-to-Noise Ratio (SNR)

range, training parameters, and key concepts. We also review DL-AMR methods not looked at in other related surveys, including those for MIMO communication systems. While our main scope remains the DL models in AMR domain, for completeness we also discuss the most relevant ML models connected to AMR field. It is worth mentioning that we differentiate between DL-AMR, which refers to the models where the features are automatically extracted from data that has a complex structure and inner correlations, and ML-AMR, where features hand-crafting is required to be done by algorithms designed by expert engineers. Hence, the main key perspectives which distinguish this paper from earlier surveys can be outlined as follows:

* We provide an inclusive review of the existing DL models in AMR domain for both SISO and MIMO systems, instead of briefly discussing the main DL models or focusing on a single type of communication system, e.g. SISO systems [8], [9].

* We particularly discuss the most recent DL models from the perspective of AMR, focusing on their applicability to this area and their structures and parameters.

To the best of our knowledge, this is the first time that AMR is inclusively reviewed for both SISO and MIMO systems in wireless communications from a DL angle.

### D. ORGANIZATION OF THE SURVEY

This paper is structured in a top-down manner as shown in Fig. 1. The outline of this paper can be summarised as:

* First, we begin, in section II, by presenting a basic background of the traditional process of FB-AMR followed by providing a basic ML and DL background to illustrate the main required concepts for the rest of this article. Thereafter, we discuss the main key factors that encourage the implementation of ML and DL models in AMR domain, aiming to clarify why such models can outperform the traditional AMR methods.

* Later in section III, a discussion on the integration of ML models into AMR methods is provided for SISO and MIMO systems.

* Since DL models require large-scale datasets, we begin section IV by presenting the specifications of some existing radio signals datasets. This provides the required information about the signal parameters and
<figure><img src="image_1.png" /><figcaption>FIGURE 1. The overall organization of the paper.</figcaption></figure>

channel characteristics of these datasets. Therefore, it will help to understand their usage in several of the presented DL-AMR methods. Then, we review the recent DL algorithms for AMR in SISO systems. We then review the existing DL-AMR methods for MIMO systems.

* Finally, we wrap up this article by providing open challenges and potential research directions along with concluding remarks in section V and in section VI, respectively.

In Fig. 1, a chart of the overall organization of the paper is provided. Furthermore, a list of main abbreviations is provided in Table 2.

## II. BACKGROUND

In order to further illustrate the integration of ML and DL models into AMR process, we begin this section by providing an overview of a typical AMR process. Afterwards, we present some basic background information on ML and DL, and then we seal up this section by clarifying why such models are needed in AMR domain.

### A. FB-AMR PROCESS

A typical AMR system consists of two subsystems: features extraction and classification subsystems. In the former subsystem, there are two phases: the pre-processing phase and the features selection phase. Fig. 2 shows a general block diagram of a features-based recognition AMR method. First, the signal will pass throw the pre-processing phase to determine several variables such as Carrier Frequency Offset (CFO), baud rate, Phase Offset (PO), SNR, and timing offsets. Next, the features will be extracted to

<table><caption>TABLE 2. List of abbreviations in alphabetical order.</caption><thead><tr><td>Acronym</td><td>Explanation</td></tr></thead><tbody><tr><td>ACGAN</td><td>Auxiliary Classification Generative Adversarial Network</td></tr><tr><td>AE</td><td>Auto-Encoder</td></tr><tr><td>AEN</td><td>Auto-Encoding Network</td></tr><tr><td>AI</td><td>Artificial Intelligence</td></tr><tr><td>AMR</td><td>Automatic Modulation Recognition</td></tr><tr><td>ANN</td><td>Artificial Neural Network</td></tr><tr><td>APoZ</td><td>Average Percentage of Zeros</td></tr><tr><td>AWGN</td><td>Additive White Gaussian Noise</td></tr><tr><td>BRNN</td><td>Bidirectional RNN</td></tr><tr><td>CAE</td><td>Convolutional AE</td></tr><tr><td>CCF</td><td>Cyclic Correntropy Function</td></tr><tr><td>CCNN</td><td>Cyclic CNN</td></tr><tr><td>CFO</td><td>Carrier Frequency Offset</td></tr><tr><td>CLDNN</td><td>Convolutional Long short-term Deep Neural Network</td></tr><tr><td>CNN</td><td>Convolutional Neural Network</td></tr><tr><td>CR</td><td>Cognitive Radio</td></tr><tr><td>CS</td><td>Cyclic Spectra</td></tr><tr><td>CSI</td><td>Channel State Information</td></tr><tr><td>CWD</td><td>Choi-Williams time-frequency Distribution</td></tr><tr><td>DBN</td><td>Deep Belief Network</td></tr><tr><td>DL</td><td>Deep Learning</td></tr><tr><td>DNN</td><td>Deep Neural Network</td></tr><tr><td>DPN</td><td>Dual Path Network</td></tr><tr><td>DRCN</td><td>Deep Reconstruction Classification Network</td></tr><tr><td>ELM</td><td>Extreme Learning Machine</td></tr><tr><td>FB</td><td>Feature-based</td></tr><tr><td>FLOCAF</td><td>Fractional Lower-Order Cyclic-Autocorrelation Function</td></tr><tr><td>FLOCS</td><td>Fractional Lower-Order Cyclic-Spectrum</td></tr><tr><td>FLOPs</td><td>Floating point Operations per Seconds</td></tr><tr><td>FPGA</td><td>Field-Programmable Gate Arrays</td></tr><tr><td>GCN</td><td>Graph Convolutional Network</td></tr><tr><td>GNU</td><td>GNU's Not Unix</td></tr><tr><td>HDNN</td><td>Hybrid DNN</td></tr><tr><td>HOC</td><td>Higher Order Cumulant</td></tr><tr><td>HOM</td><td>Higher Order Moment</td></tr><tr><td>IQ</td><td>In-Phase and Quadrature</td></tr><tr><td>LB</td><td>Likelihood-Based</td></tr><tr><td>LOS</td><td>Line-Of-Sight</td></tr><tr><td>LRN</td><td>Layered Resnet Network</td></tr><tr><td>LRR</td><td>Low-Rank Representation</td></tr><tr><td>LSTM</td><td>Long Short-Term Memory</td></tr><tr><td>M-ASK</td><td>M-ary Amplitude Shift Keying</td></tr><tr><td>MIMO</td><td>Multiple-Input Multiple-Output</td></tr><tr><td>ML</td><td>Machine Learning</td></tr><tr><td>MLP</td><td>Multilayer Perceptron</td></tr><tr><td>M-PSK</td><td>M-ary phase-shift keying</td></tr><tr><td>M-QAM</td><td>M-Ary Quadrature Amplitude Modulation</td></tr><tr><td>NiN</td><td>Network-in-Network</td></tr><tr><td>NLOS</td><td>Non-Line-Of-Sight</td></tr><tr><td>OFDM</td><td>Orthogonal Frequency-Division Multiplexing</td></tr><tr><td>OSNR</td><td>Optical SNR</td></tr><tr><td>PCA</td><td>Principle Component Analysis</td></tr><tr><td>PO</td><td>Phase Offset</td></tr><tr><td>PReLU</td><td>Parametric ReLU</td></tr><tr><td>PSO</td><td>Particle Swarm Optimization</td></tr><tr><td>RBM</td><td>Restricted Boltzmann Machine</td></tr><tr><td>RNN</td><td>Recurrent Neural Network</td></tr><tr><td>SCAE</td><td>Stacked Convolutional Auto-Encoder</td></tr><tr><td>SCF</td><td>Spectral Correlation Function</td></tr><tr><td>SELU</td><td>Scaled Exponential Linear Unit</td></tr><tr><td>SGD</td><td>Stochastic Gradient Descent</td></tr><tr><td>SISO</td><td>Single-Input Single-Output</td></tr><tr><td>SNR</td><td>Signal-to-Noise Ratio</td></tr><tr><td>SSAE</td><td>Stacked Sparse Auto-Encoder</td></tr><tr><td>SSCCF</td><td>Spatial Sign Cyclic Correlation Function</td></tr><tr><td>STFT</td><td>Short-Time Fourier transform</td></tr><tr><td>SVM</td><td>Support Vector Machine</td></tr><tr><td>SW-SCMA</td><td>Sliding Window Simplified Constant Modulus Algorithm</td></tr><tr><td>Tanh</td><td>hyperbolic tangent</td></tr><tr><td>TL</td><td>Transfer Learning</td></tr><tr><td>ZF</td><td>Zero-Forcing</td></tr></tbody></table>
<figure><img src="image_2.png" /><figcaption>FIGURE 1. The overall organization of the paper.</figcaption></figure>

channel characteristics of these datasets. Therefore, it will help to understand their usage in several of the presented DL-AMR methods. Then, we review the recent DL algorithms for AMR in SISO systems. We then review the existing DL-AMR methods for MIMO systems.

* Finally, we wrap up this article by providing open challenges and potential research directions along with concluding remarks in section V and in section VI, respectively.

In Fig. 1, a chart of the overall organization of the paper is provided. Furthermore, a list of main abbreviations is provided in Table 2.

## II. BACKGROUND

In order to further illustrate the integration of ML and DL models into AMR process, we begin this section by providing an overview of a typical AMR process. Afterwards, we present some basic background information on ML and DL, and then we seal up this section by clarifying why such models are needed in AMR domain.

### A. FB-AMR PROCESS

A typical AMR system consists of two subsystems: features extraction and classification subsystems. In the former subsystem, there are two phases: the pre-processing phase and the features selection phase. Fig. 2 shows a general block diagram of a features-based recognition AMR method. First, the signal will pass throw the pre-processing phase to determine several variables such as Carrier Frequency Offset (CFO), baud rate, Phase Offset (PO), SNR, and timing offsets. Next, the features will be extracted to

<table><caption>TABLE 2. List of abbreviations in alphabetical order.</caption><thead><tr><td>Acronym</td><td>Explanation</td></tr></thead><tbody><tr><td>ACGAN</td><td>Auxiliary Classification Generative Adversarial Network</td></tr><tr><td>AE</td><td>Auto-Encoder</td></tr><tr><td>AEN</td><td>Auto-Encoding Network</td></tr><tr><td>AI</td><td>Artificial Intelligence</td></tr><tr><td>AMR</td><td>Automatic Modulation Recognition</td></tr><tr><td>ANN</td><td>Artificial Neural Network</td></tr><tr><td>APoZ</td><td>Average Percentage of Zeros</td></tr><tr><td>AWGN</td><td>Additive White Gaussian Noise</td></tr><tr><td>BRNN</td><td>Bidirectional RNN</td></tr><tr><td>CAE</td><td>Convolutional AE</td></tr><tr><td>CCF</td><td>Cyclic Correntropy Function</td></tr><tr><td>CCNN</td><td>Cyclic CNN</td></tr><tr><td>CFO</td><td>Carrier Frequency Offset</td></tr><tr><td>CLDNN</td><td>Convolutional Long short-term Deep Neural Network</td></tr><tr><td>CNN</td><td>Convolutional Neural Network</td></tr><tr><td>CR</td><td>Cognitive Radio</td></tr><tr><td>CS</td><td>Cyclic Spectra</td></tr><tr><td>CSI</td><td>Channel State Information</td></tr><tr><td>CWD</td><td>Choi-Williams time-frequency Distribution</td></tr><tr><td>DBN</td><td>Deep Belief Network</td></tr><tr><td>DL</td><td>Deep Learning</td></tr><tr><td>DNN</td><td>Deep Neural Network</td></tr><tr><td>DPN</td><td>Dual Path Network</td></tr><tr><td>DRCN</td><td>Deep Reconstruction Classification Network</td></tr><tr><td>ELM</td><td>Extreme Learning Machine</td></tr><tr><td>FB</td><td>Feature-based</td></tr><tr><td>FLOCAF</td><td>Fractional Lower-Order Cyclic-Autocorrelation Function</td></tr><tr><td>FLOCS</td><td>Fractional Lower-Order Cyclic-Spectrum</td></tr><tr><td>FLOPs</td><td>Floating point Operations per Seconds</td></tr><tr><td>FPGA</td><td>Field-Programmable Gate Arrays</td></tr><tr><td>GCN</td><td>Graph Convolutional Network</td></tr><tr><td>GNU</td><td>GNU's Not Unix</td></tr><tr><td>HDNN</td><td>Hybrid DNN</td></tr><tr><td>HOC</td><td>Higher Order Cumulant</td></tr><tr><td>HOM</td><td>Higher Order Moment</td></tr><tr><td>IQ</td><td>In-Phase and Quadrature</td></tr><tr><td>LB</td><td>Likelihood-Based</td></tr><tr><td>LOS</td><td>Line-Of-Sight</td></tr><tr><td>LRN</td><td>Layered Resnet Network</td></tr><tr><td>LRR</td><td>Low-Rank Representation</td></tr><tr><td>LSTM</td><td>Long Short-Term Memory</td></tr><tr><td>M-ASK</td><td>M-ary Amplitude Shift Keying</td></tr><tr><td>MIMO</td><td>Multiple-Input Multiple-Output</td></tr><tr><td>ML</td><td>Machine Learning</td></tr><tr><td>MLP</td><td>Multilayer Perceptron</td></tr><tr><td>M-PSK</td><td>M-ary phase-shift keying</td></tr><tr><td>M-QAM</td><td>M-Ary Quadrature Amplitude Modulation</td></tr><tr><td>NiN</td><td>Network-in-Network</td></tr><tr><td>NLOS</td><td>Non-Line-Of-Sight</td></tr><tr><td>OFDM</td><td>Orthogonal Frequency-Division Multiplexing</td></tr><tr><td>OSNR</td><td>Optical SNR</td></tr><tr><td>PCA</td><td>Principle Component Analysis</td></tr><tr><td>PO</td><td>Phase Offset</td></tr><tr><td>PReLU</td><td>Parametric ReLU</td></tr><tr><td>PSO</td><td>Particle Swarm Optimization</td></tr><tr><td>RBM</td><td>Restricted Boltzmann Machine</td></tr><tr><td>RNN</td><td>Recurrent Neural Network</td></tr><tr><td>SCAE</td><td>Stacked Convolutional Auto-Encoder</td></tr><tr><td>SCF</td><td>Spectral Correlation Function</td></tr><tr><td>SELU</td><td>Scaled Exponential Linear Unit</td></tr><tr><td>SGD</td><td>Stochastic Gradient Descent</td></tr><tr><td>SISO</td><td>Single-Input Single-Output</td></tr><tr><td>SNR</td><td>Signal-to-Noise Ratio</td></tr><tr><td>SSAE</td><td>Stacked Sparse Auto-Encoder</td></tr><tr><td>SSCCF</td><td>Spatial Sign Cyclic Correlation Function</td></tr><tr><td>STFT</td><td>Short-Time Fourier transform</td></tr><tr><td>SVM</td><td>Support Vector Machine</td></tr><tr><td>SW-SCMA</td><td>Sliding Window Simplified Constant Modulus Algorithm</td></tr><tr><td>Tanh</td><td>hyperbolic tangent</td></tr><tr><td>TL</td><td>Transfer Learning</td></tr><tr><td>ZF</td><td>Zero-Forcing</td></tr></tbody></table>
<figure><img src="image_2.png" /><figcaption>FIGURE 2. Block diagram of the Feature-based AMR method.</figcaption></figure>

be then fed into the classification subsystem. Specifically, the extracted features can be categorized as follows: instantaneous time features [11], wavelet features [12], statistical features [13]–[15]. Ultimately, the decision of the modulation scheme is made by a specific classifier. Classifiers can be categorized into three main categories: (1) traditional classifiers such as classification tree, (2) ML-based classifiers, and (3) DL-based classifiers.

## B. ML AND DL OVERVIEW

Fundamentally, ML is an area of Artificial Intelligence (AI) in which the algorithms can parse data, learn from them and apply whatever they learned to make informed decisions. In fact, DL is a sophisticated evolution subset of ML which gives machines the ability to make decisions without the intervention of humans. Technically, DL is ML and functions in a similar way but they have different capabilities and provide a different interpretation of the data they convey. Moreover, DL models can learn knowledge from raw data and decide if a prediction is accurate or not without any guidance. Furthermore, a key difference between traditional ML and DL is in the features extraction mechanism. ML methods apply the learning algorithms on hand-crafted engineering features, while in DL models, the features are learned automatically in multiple levels [10]. However, ML and DL approaches can be categorized as follows: Supervised, semi-supervised and unsupervised approaches [10]. Fig. 3 shows the Venn diagram of the relation between AI, ML, and DL.

### 1) DEEP NEURAL NETWORK (DNN) OVERVIEW

DL models are designed based on a specific layered structure of Artificial Neural Networks (ANNs). The idea behind the design of ANN is inspired by the biological neural network of the human brain. Fig. 4 shows the structure of a simple ANN. The first layer is called the input layer which receives the input data, while the rightmost one is the output layer which returns the output data. The layers between these two layers perform mathematical computations on the input data and they are called hidden layers since their values aren't observable in the training set. In general, an ANN with several hidden layers is referred to as a DNN and the more hidden layers it has, the deeper it is.

Each layer comprises several nodes called neurons. The neuron is the basic element of a neural network. It receives an input x, processes it, and generates an output. This output is either the final output or it will be sent to other neurons, through weighted connections, for further processing.

<figure><img src="image_3.png" /><figcaption>FIGURE 3. The taxonomy of AI as the broader umbrella for both ML and DL, and DL as a sub-branch of ML.</figcaption></figure>

Each connection between these neurons is associated with a specific weight $W_i$. These weights are randomly initialized and are updated during the model training process. Each weight dictates the importance of the associated input value. Besides, another linear component, called the bias $b_i$, is applied to the input. The bias role is to change the range of the weight multiplied input. As can be seen, the final linear component is:

$$L_i = x_i \times W_i + b_i \quad (1)$$

Subsequently, a non-linear function, called activation function $f$, is applied to the combination of all linear components of a specific neuron. Thereafter, the output $y_i$ will be look like:

$$y_i = f \left( \sum_i L_i \right) = f \left( \sum_i x_i W_i + b_i \right) \quad (2)$$

As shown in Table 3, there are several activation functions such as Sigmoid, hyperbolic tangent (Tanh), Rectified Linear Unit (ReLU) [16], Parametric ReLU (PReLU), Scaled Exponential Linear Unit (SELU) [17] and Softmax [10], [18].

However, the main aim when training a DNN is to optimize the prediction accuracy and, thus to minimize the prediction error. Therefore, a cost or loss function should be used for measuring this error and react accordingly. Besides, the neural network should be trained on random equal-sized
<figure><img src="image_3.png" /><figcaption>FIGURE 2. Block diagram of the Feature-based AMR method.</figcaption></figure>

be then fed into the classification subsystem. Specifically, the extracted features can be categorized as follows: instantaneous time features [11], wavelet features [12], statistical features [13]–[15]. Ultimately, the decision of the modulation scheme is made by a specific classifier. Classifiers can be categorized into three main categories: (1) traditional classifiers such as classification tree, (2) ML-based classifiers, and (3) DL-based classifiers.

## B. ML AND DL OVERVIEW

Fundamentally, ML is an area of Artificial Intelligence (AI) in which the algorithms can parse data, learn from them and apply whatever they learned to make informed decisions. In fact, DL is a sophisticated evolution subset of ML which gives machines the ability to make decisions without the intervention of humans. Technically, DL is ML and functions in a similar way but they have different capabilities and provide a different interpretation of the data they convey. Moreover, DL models can learn knowledge from raw data and decide if a prediction is accurate or not without any guidance. Furthermore, a key difference between traditional ML and DL is in the features extraction mechanism. ML methods apply the learning algorithms on hand-crafted engineering features, while in DL models, the features are learned automatically in multiple levels [10]. However, ML and DL approaches can be categorized as follows: Supervised, semi-supervised and unsupervised approaches [10]. Fig. 3 shows the Venn diagram of the relation between AI, ML, and DL.

### 1) DEEP NEURAL NETWORK (DNN) OVERVIEW

DL models are designed based on a specific layered structure of Artificial Neural Networks (ANNs). The idea behind the design of ANN is inspired by the biological neural network of the human brain. Fig. 4 shows the structure of a simple ANN. The first layer is called the input layer which receives the input data, while the rightmost one is the output layer which returns the output data. The layers between these two layers perform mathematical computations on the input data and they are called hidden layers since their values aren't observable in the training set. In general, an ANN with several hidden layers is referred to as a DNN and the more hidden layers it has, the deeper it is.

Each layer comprises several nodes called neurons. The neuron is the basic element of a neural network. It receives an input x, processes it, and generates an output. This output is either the final output or it will be sent to other neurons, through weighted connections, for further processing.

<figure><img src="image_4.png" /><figcaption>FIGURE 3. The taxonomy of AI as the broader umbrella for both ML and DL, and DL as a sub-branch of ML.</figcaption></figure>

Each connection between these neurons is associated with a specific weight $W_i$. These weights are randomly initialized and are updated during the model training process. Each weight dictates the importance of the associated input value. Besides, another linear component, called the bias $b_i$, is applied to the input. The bias role is to change the range of the weight multiplied input. As can be seen, the final linear component is:

$$L_i = x_i \times W_i + b_i \quad (1)$$

Subsequently, a non-linear function, called activation function $f$, is applied to the combination of all linear components of a specific neuron. Thereafter, the output $y_i$ will be look like:

$$y_i = f \left( \sum_i L_i \right) = f \left( \sum_i x_i W_i + b_i \right) \quad (2)$$

As shown in Table 3, there are several activation functions such as Sigmoid, hyperbolic tangent (Tanh), Rectified Linear Unit (ReLU) [16], Parametric ReLU (PReLU), Scaled Exponential Linear Unit (SELU) [17] and Softmax [10], [18].

However, the main aim when training a DNN is to optimize the prediction accuracy and, thus to minimize the prediction error. Therefore, a cost or loss function should be used for measuring this error and react accordingly. Besides, the neural network should be trained on random equal-sized
<figure><img src="image_4.png" /><figcaption>FIGURE 4. A simple Artificial Neural Network (ANN).</figcaption></figure>

<table><caption>TABLE 3. The mathematical representations of activation functions.</caption><thead><tr><th>Name</th><th>Equation</th><th></th></tr></thead><tbody><tr><td>sigmoid</td><td>sigmoid(x) = <sup>1</sup>&frasl;<sub>1 + e<sup>-x</sup></sub></td><td>(3)</td></tr><tr><td>ReLU</td><td>ReLU(x) = <span style="font-size: 2em; vertical-align: middle;">{</span><table style="display: inline-table; vertical-align: middle; border-collapse: collapse; margin: 0 0.2em;"><tbody><tr><td style="border: none; padding: 0 0.2em; text-align: center;">x</td><td style="border: none; padding: 0 0.2em; text-align: center;">x &gt; 0</td></tr><tr><td style="border: none; padding: 0 0.2em; text-align: center;">0</td><td style="border: none; padding: 0 0.2em; text-align: center;">x &le; 0</td></tr></tbody></table><span style="font-size: 2em; vertical-align: middle;">}</span></td><td>(4)</td></tr><tr><td>PReLU</td><td>PReLU(x) = <span style="font-size: 2em; vertical-align: middle;">{</span><table style="display: inline-table; vertical-align: middle; border-collapse: collapse; margin: 0 0.2em;"><tbody><tr><td style="border: none; padding: 0 0.2em; text-align: center;">x</td><td style="border: none; padding: 0 0.2em; text-align: center;">x &gt; 0</td></tr><tr><td style="border: none; padding: 0 0.2em; text-align: center;">ax</td><td style="border: none; padding: 0 0.2em; text-align: center;">x &le; 0</td></tr></tbody></table><span style="font-size: 2em; vertical-align: middle;">}</span></td><td>(5)</td></tr><tr><td>Tanh</td><td>tanh(x) = <sup>e<sup>x</sup> - e<sup>-x</sup></sup>&frasl;<sub>e<sup>x</sup> + e<sup>-x</sup></sub></td><td>(6)</td></tr><tr><td>SELU</td><td>SELU(x) = &lambda; <span style="font-size: 2em; vertical-align: middle;">{</span><table style="display: inline-table; vertical-align: middle; border-collapse: collapse; margin: 0 0.2em;"><tbody><tr><td style="border: none; padding: 0 0.2em; text-align: center;">x</td><td style="border: none; padding: 0 0.2em; text-align: center;">if x &gt; 0</td></tr><tr><td style="border: none; padding: 0 0.2em; text-align: center;">&alpha;e<sup>x</sup> - &alpha;</td><td style="border: none; padding: 0 0.2em; text-align: center;">if x &le; 0</td></tr></tbody></table><span style="font-size: 2em; vertical-align: middle;">}</span> where &alpha; and &lambda; are pre-defined constants</td><td>(7)</td></tr><tr><td>Softmax</td><td>softmax(x<sub>i</sub>) = <sup>e<sup>x<sub>i</sub></sup></sup>&frasl;<sub>&sum;<sub>j=0</sub><sup>k</sup> e<sup>x<sub>j</sub></sup></sub></td><td>(8)</td></tr></tbody></table>

batches which makes the model more generalized than sending the whole input in one go. Moreover, optimization and regularization techniques can be adopted such as dropout regularization and gradient descent optimization algorithms. It is worth mentioning that Multi-Layer Perceptron (MLP) is always feed-forward, while DNN can have loops.

## 2) CNN OVERVIEW

CNN is a DL algorithm that comprises several types of layers, and has several advantages over DNNs. CNN is much more

like the human visual processing system, especially when it comes to multi-dimensional inputs (e.g. 2D and 3D images). Most of all, the parameters of CNNs are significantly fewer than a fully connected network of similar size, and suffer less from the diminishing gradient problem. Fig. 5 shows an example of CNN structure. Mainly, the architecture of CNN consists of a combination of several types of layers. These layers can be summarized as follows [10], [18]:

* Convolutional Layer: This layer is responsible for convolving feature maps from previous layers with learnable kernels. Thereafter, a linear or non-linear activation function is applied to the output of these kernels to generate an activation map as its output.

* Rectification Layer: This layer performs element-wise absolute value operation on the input value.

* Pooling Layer: This layer takes the activation map generated by the non-linearity layer, and performs a down sampling operation in order to reduce its size.

* Fully Connected or Dense Layer: This layer is a MLP. It is responsible for mapping the activation maps from previous layers into a class probability distribution.

* Dropout Layer: This layer is practically used to prevent over-fitting in the training process by using Dropout along with other techniques such as L2 Regularization.

* Softmax Layer: This layer uses Softmax function in order to computes the score of each class over all possible classes. This function is usually used in the classification layer.

However, the total number of learnable parameters is an important metric to measure the complexity and the required memory of a CNN model.

## 3) RECURRENT NEURAL NETWORK (RNN) OVERVIEW

RNN is simply a generalized feed-forward ANN that has a memory to store the output it has already learned from the previous input. As shown in Fig. 6, RNN performs the same function for every input with a consideration of the last computed value. However, traditional ANNs approaches, DNNs, and CNNs deal only with fixed-length input. On the contrary, RNNs can handle a sequence of vectors over time. Moreover, a modified version of RNN, called Long Short-Term Memory (LSTM), is able to not only solve the problem of vanishing gradient, but also remember past data easily. LSTM is trained using back-propagation and can classify a time series given time lags of unknown duration. It is worth mentioning that there are several modified versions of RNN in the literature such as convolutional LSTM [18].

## 4) AUTO-ENCODER (AE) OVERVIEW

AE is practically a DNN approach which can learn features unsupervised based on efficient data encoding and decoding. As shown in Fig. 7, the inputs are encoded and mapped in the lower dimensional features space with a constructive feature representation. Thereafter, the same process is repeated till the desired feature dimensional space is reached. In the
<figure><img src="image_5.png" /><figcaption>FIGURE 4. A simple Artificial Neural Network (ANN).</figcaption></figure>

<table><caption>TABLE 3. The mathematical representations of activation functions.</caption><thead><tr><th>Name</th><th>Equation</th><th></th></tr></thead><tbody><tr><td>sigmoid</td><td>sigmoid(x) = <sup>1</sup>&frasl;<sub>1 + e<sup>-x</sup></sub></td><td>(3)</td></tr><tr><td>ReLU</td><td>ReLU(x) = <span style="font-size: 2em; vertical-align: middle;">{</span><table style="display: inline-table; vertical-align: middle; border-collapse: collapse; margin: 0 0.2em;"><tbody><tr><td style="border: none; padding: 0 0.2em; text-align: center;">x</td><td style="border: none; padding: 0 0.2em; text-align: center;">x &gt; 0</td></tr><tr><td style="border: none; padding: 0 0.2em; text-align: center;">0</td><td style="border: none; padding: 0 0.2em; text-align: center;">x &le; 0</td></tr></tbody></table><span style="font-size: 2em; vertical-align: middle;">}</span></td><td>(4)</td></tr><tr><td>PReLU</td><td>PReLU(x) = <span style="font-size: 2em; vertical-align: middle;">{</span><table style="display: inline-table; vertical-align: middle; border-collapse: collapse; margin: 0 0.2em;"><tbody><tr><td style="border: none; padding: 0 0.2em; text-align: center;">x</td><td style="border: none; padding: 0 0.2em; text-align: center;">x &gt; 0</td></tr><tr><td style="border: none; padding: 0 0.2em; text-align: center;">ax</td><td style="border: none; padding: 0 0.2em; text-align: center;">x &le; 0</td></tr></tbody></table><span style="font-size: 2em; vertical-align: middle;">}</span></td><td>(5)</td></tr><tr><td>Tanh</td><td>tanh(x) = <sup>e<sup>x</sup> - e<sup>-x</sup></sup>&frasl;<sub>e<sup>x</sup> + e<sup>-x</sup></sub></td><td>(6)</td></tr><tr><td>SELU</td><td>SELU(x) = &lambda; <span style="font-size: 2em; vertical-align: middle;">{</span><table style="display: inline-table; vertical-align: middle; border-collapse: collapse; margin: 0 0.2em;"><tbody><tr><td style="border: none; padding: 0 0.2em; text-align: center;">x</td><td style="border: none; padding: 0 0.2em; text-align: center;">if x &gt; 0</td></tr><tr><td style="border: none; padding: 0 0.2em; text-align: center;">&alpha;e<sup>x</sup> - &alpha;</td><td style="border: none; padding: 0 0.2em; text-align: center;">if x &le; 0</td></tr></tbody></table><span style="font-size: 2em; vertical-align: middle;">}</span> where &alpha; and &lambda; are pre-defined constants</td><td>(7)</td></tr><tr><td>Softmax</td><td>softmax(x<sub>i</sub>) = <sup>e<sup>x<sub>i</sub></sup></sup>&frasl;<sub>&sum;<sub>j=0</sub><sup>k</sup> e<sup>x<sub>j</sub></sup></sub></td><td>(8)</td></tr></tbody></table>

batches which makes the model more generalized than sending the whole input in one go. Moreover, optimization and regularization techniques can be adopted such as dropout regularization and gradient descent optimization algorithms. It is worth mentioning that Multi-Layer Perceptron (MLP) is always feed-forward, while DNN can have loops.

## 2) CNN OVERVIEW

CNN is a DL algorithm that comprises several types of layers, and has several advantages over DNNs. CNN is much more

like the human visual processing system, especially when it comes to multi-dimensional inputs (e.g. 2D and 3D images). Most of all, the parameters of CNNs are significantly fewer than a fully connected network of similar size, and suffer less from the diminishing gradient problem. Fig. 5 shows an example of CNN structure. Mainly, the architecture of CNN consists of a combination of several types of layers. These layers can be summarized as follows [10], [18]:

* Convolutional Layer: This layer is responsible for convolving feature maps from previous layers with learnable kernels. Thereafter, a linear or non-linear activation function is applied to the output of these kernels to generate an activation map as its output.

* Rectification Layer: This layer performs element-wise absolute value operation on the input value.

* Pooling Layer: This layer takes the activation map generated by the non-linearity layer, and performs a down sampling operation in order to reduce its size.

* Fully Connected or Dense Layer: This layer is a MLP. It is responsible for mapping the activation maps from previous layers into a class probability distribution.

* Dropout Layer: This layer is practically used to prevent over-fitting in the training process by using Dropout along with other techniques such as L2 Regularization.

* Softmax Layer: This layer uses Softmax function in order to computes the score of each class over all possible classes. This function is usually used in the classification layer.

However, the total number of learnable parameters is an important metric to measure the complexity and the required memory of a CNN model.

## 3) RECURRENT NEURAL NETWORK (RNN) OVERVIEW

RNN is simply a generalized feed-forward ANN that has a memory to store the output it has already learned from the previous input. As shown in Fig. 6, RNN performs the same function for every input with a consideration of the last computed value. However, traditional ANNs approaches, DNNs, and CNNs deal only with fixed-length input. On the contrary, RNNs can handle a sequence of vectors over time. Moreover, a modified version of RNN, called Long Short-Term Memory (LSTM), is able to not only solve the problem of vanishing gradient, but also remember past data easily. LSTM is trained using back-propagation and can classify a time series given time lags of unknown duration. It is worth mentioning that there are several modified versions of RNN in the literature such as convolutional LSTM [18].

## 4) AUTO-ENCODER (AE) OVERVIEW

AE is practically a DNN approach which can learn features unsupervised based on efficient data encoding and decoding. As shown in Fig. 7, the inputs are encoded and mapped in the lower dimensional features space with a constructive feature representation. Thereafter, the same process is repeated till the desired feature dimensional space is reached. In the
<figure><img src="image_5.png" /><figcaption>FIGURE 5. Structure example of CNN.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>FIGURE 6. Structure example of RNN.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>FIGURE 7. Structure example of AE.</figcaption></figure>

decoding phase, the actual features are obtained from lower dimensional ones with reverse processing. Several kinds of AE exist in the literature such as variational AEs and split-brain AEs [10], [18].

### 5) RESTRICTED BOLTZMANN MACHINE (RBM) OVERVIEW

A RBM is also trained in an unsupervised manner. RBM consists of a visible layer, a hidden layer, and weighted connections between them. Each neuron in RBM is a stochastic

binary unit. The neurons in the visible layer are mutually independent given the hidden ones, and *vice versa* [18]. As in a simple feed forward ANN, the visible layer receives the input data, and the neurons of the hidden layer are fully connected to all neurons in the visible layer. However, the state of the neurons in the hidden layer can affect those in the visible one, and *vice versa*.

The architecture of a Deep Belief Network (DBN) [18] can be formed by stacking RBMs or AEs. The process in a DBN consists of two main stages: (1) pre-training stage and (2) fine-tuning stage. The former stage is performed in each layer of RBM in order to initialize the parameters of the DBN. In the latter stage, regression algorithms such as softmax regression can be applied to provide error prediction and, hence, optimize the parameters by back-propagation algorithm. Fig. 8 shows the general structure of a DBN.

<figure><img src="image_8.png" /><figcaption>FIGURE 8. General structure of DBN.</figcaption></figure>

#### a: THE NEED OF ML AND DL IN FB-AMR

The performance of the conventional FB-AMR methods heavily depends on the extracted features from the received signals and requires the supervision of expert engineers. Moreover, a set of different features has to be built manually in order to recognize different modulation schemes. Besides, conventional FB methods require building a classification tree with manual thresholds in order to decide on the modulation scheme. In such methods, the AMR process may
<figure><img src="image_6.png" /><figcaption>FIGURE 5. Structure example of CNN.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>FIGURE 6. Structure example of RNN.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>FIGURE 7. Structure example of AE.</figcaption></figure>

decoding phase, the actual features are obtained from lower dimensional ones with reverse processing. Several kinds of AE exist in the literature such as variational AEs and split-brain AEs [10], [18].

### 5) RESTRICTED BOLTZMANN MACHINE (RBM) OVERVIEW

A RBM is also trained in an unsupervised manner. RBM consists of a visible layer, a hidden layer, and weighted connections between them. Each neuron in RBM is a stochastic

binary unit. The neurons in the visible layer are mutually independent given the hidden ones, and *vice versa* [18]. As in a simple feed forward ANN, the visible layer receives the input data, and the neurons of the hidden layer are fully connected to all neurons in the visible layer. However, the state of the neurons in the hidden layer can affect those in the visible one, and *vice versa*.

The architecture of a Deep Belief Network (DBN) [18] can be formed by stacking RBMs or AEs. The process in a DBN consists of two main stages: (1) pre-training stage and (2) fine-tuning stage. The former stage is performed in each layer of RBM in order to initialize the parameters of the DBN. In the latter stage, regression algorithms such as softmax regression can be applied to provide error prediction and, hence, optimize the parameters by back-propagation algorithm. Fig. 8 shows the general structure of a DBN.

<figure><img src="image_9.png" /><figcaption>FIGURE 8. General structure of DBN.</figcaption></figure>

#### a: THE NEED OF ML AND DL IN FB-AMR

The performance of the conventional FB-AMR methods heavily depends on the extracted features from the received signals and requires the supervision of expert engineers. Moreover, a set of different features has to be built manually in order to recognize different modulation schemes. Besides, conventional FB methods require building a classification tree with manual thresholds in order to decide on the modulation scheme. In such methods, the AMR process may
need the knowledge of SNR value especially in the case of wide range SNR applications, then SNR estimation is also required. However, the hand-crafted features may have different values under varying noise conditions and, thus, different classification trees should be built or feature selection algorithms should be designed to build a robust set of hand-crafted features [19].

That is why ML and DL approaches have been developed to reduce tedious threshold operations and to improve the overall performance. DL approaches can eliminate the SNR estimation requirement since they can learn automatically optimal features, under varying noise regimes, without any need for precisely designed features. Moreover, DL models can be extended to several applications by adopting the transfer learning approach for example. Recently, many researchers proposed incorporating ML and DL models into FB-AMR methods to explore different types of features. This integration allows an automated features extraction process by employing different architectures of DNNs. However, self-learned features can be more relevant and, hence, achieve better performance.

### III. MACHINE LEARNING IN AMR DOMAIN FOR WIRELESS COMMUNICATIONS

ML is a subset of AI that exploits statistical learning algorithms for designing and implementing intelligent systems. Many existing works employed ML classifiers in order to improve the classification accuracy of AMR for both SISO and MIMO systems. Numerous researches were proposed based on ANNs [15], [20]–[22], Extreme Learning Machine (ELM) [23]–[25] as well as Support Vector Machine (SVM) [26], [27]. In this section, the presented ML classifiers are considered to be different from those in the DL subset. First, we review the recent ML-AMR methods in SISO systems. Then, we discuss the most relevant ML models in AMR for MIMO systems. Finally, we conclude this section with a brief discussion and learned lessons.

#### A. ML-AMR METHODS FOR SISO SYSTEMS

In [24], an ELM-based AMR method was proposed by using one dimension local binary pattern [28] for features extraction. The classification accuracy was investigated under known channel states where the PO and CFO were recovered prior to modulation classification. Moreover, the overall performance was studied under several receiver conditions such as the symbols number, timing offset, PO, impulsive noise, and Doppler shift. In [25], Gabor filter and Cuckoo search algorithm were employed for features extraction and optimization, respectively. However, the decision on the modulation scheme was made by ELM classifier. In [26], a low-complexity method for AMR and SNR estimation was introduced over multipath fading channels. Principal Component Analysis (PCA) was adopted for features extraction, while two different types of SVMs were used for classification and SNR estimation. In particular, three models of support vector classifiers were trained to differentiate a modulation

type from the others. K-means clustering was used in [27] to extract the required features to feed the SVM classifier. The authors in [29] presented a compressive sensing AMR method by employing the higher-order cyclic cumulants and SVM classifier. SNR variations were considered in [19] in order to design a robust SVM-based AMR method. Hence, the normalized-centered variance was used to select the noise-insensitive features from a large set of extracted features. Besides, attribute reduction based on rough set theory was applied to remove all possible redundant features.

Wireless communication systems may be subject to non-Gaussian interference and impulsive noise. Therefore, the communication channel can be better modeled by heavy-tailed distributions, such as the non-Gaussian $\alpha$-stable one. However, there are several promising solutions for AMR in the presence of impulsive non-Gaussian noise such as the Fractional Lower-Order Cyclic-Spectrum (FLOCS), the Spatial Sign Cyclic Correlation Function (SSCCF), the Fractional Lower-Order Cyclic Autocorrelation Function (FLOCAF), and the Cyclic Correntropy Function (CCF). The authors in [30] employed the graph-based FLOCS analysis in the $\alpha$-stable impulsive noise environment for AMR. Firstly, the FLOCS of the received signal was obtained by the transformation of its fractional lower-order moments. Then, the modulation scheme was identified using the graph-based AMR mechanism by employing the extracted features from the adjacency matrices corresponding to the graph representation of FLOCS. In comparison, FLOCAF and CCF were adopted in [31], and a mathematical expression demonstrating that SSCCF is a particular case of the FLOCAF was derived. Besides, a detailed analysis of the cyclic spectrum of BPSK, QPSK, 8QAM, 16QAM, and 32QAM signals obtained by the FLOCAF and CCF was provided. Moreover, both FLOCAF and CCF were found to allow the symbol rate parameter estimation, where the latter was found to be more efficient.

#### B. ML-AMR METHODS FOR MIMO SYSTEMS

The ANN-based AMR algorithm in [20] was proposed for MIMO systems, and it was then modified in [21] by considering the time variation in the channel over an observation interval. A sliding window simplified constant modulus algorithm (SW-SCMA) was used during the blind channel estimation phase. The work reported in [21] considered a more realistic scenario because the channel matrix is unknown and blindly estimated by SCMA without any channel compensation. In [22], three AMR algorithms were presented. In the first one, the modulation schemes were directly recognized without any channel equalization. The second algorithm assumes that the channel matrix is perfectly known at the receiver side and it uses the Zero-Forcing (ZF) equalization to recover the transmitted signal symbols before features extraction. In the last one, the receiver has no prior knowledge of the channel matrix and it is blindly estimated by SCMA likewise in [20]. It is worth mentioning that the final decision was made using a majority rule by fusing all individual decisions.
need the knowledge of SNR value especially in the case of wide range SNR applications, then SNR estimation is also required. However, the hand-crafted features may have different values under varying noise conditions and, thus, different classification trees should be built or feature selection algorithms should be designed to build a robust set of hand-crafted features [19].

That is why ML and DL approaches have been developed to reduce tedious threshold operations and to improve the overall performance. DL approaches can eliminate the SNR estimation requirement since they can learn automatically optimal features, under varying noise regimes, without any need for precisely designed features. Moreover, DL models can be extended to several applications by adopting the transfer learning approach for example. Recently, many researchers proposed incorporating ML and DL models into FB-AMR methods to explore different types of features. This integration allows an automated features extraction process by employing different architectures of DNNs. However, self-learned features can be more relevant and, hence, achieve better performance.

### III. MACHINE LEARNING IN AMR DOMAIN FOR WIRELESS COMMUNICATIONS

ML is a subset of AI that exploits statistical learning algorithms for designing and implementing intelligent systems. Many existing works employed ML classifiers in order to improve the classification accuracy of AMR for both SISO and MIMO systems. Numerous researches were proposed based on ANNs [15], [20]–[22], Extreme Learning Machine (ELM) [23]–[25] as well as Support Vector Machine (SVM) [26], [27]. In this section, the presented ML classifiers are considered to be different from those in the DL subset. First, we review the recent ML-AMR methods in SISO systems. Then, we discuss the most relevant ML models in AMR for MIMO systems. Finally, we conclude this section with a brief discussion and learned lessons.

#### A. ML-AMR METHODS FOR SISO SYSTEMS

In [24], an ELM-based AMR method was proposed by using one dimension local binary pattern [28] for features extraction. The classification accuracy was investigated under known channel states where the PO and CFO were recovered prior to modulation classification. Moreover, the overall performance was studied under several receiver conditions such as the symbols number, timing offset, PO, impulsive noise, and Doppler shift. In [25], Gabor filter and Cuckoo search algorithm were employed for features extraction and optimization, respectively. However, the decision on the modulation scheme was made by ELM classifier. In [26], a low-complexity method for AMR and SNR estimation was introduced over multipath fading channels. Principal Component Analysis (PCA) was adopted for features extraction, while two different types of SVMs were used for classification and SNR estimation. In particular, three models of support vector classifiers were trained to differentiate a modulation

type from the others. K-means clustering was used in [27] to extract the required features to feed the SVM classifier. The authors in [29] presented a compressive sensing AMR method by employing the higher-order cyclic cumulants and SVM classifier. SNR variations were considered in [19] in order to design a robust SVM-based AMR method. Hence, the normalized-centered variance was used to select the noise-insensitive features from a large set of extracted features. Besides, attribute reduction based on rough set theory was applied to remove all possible redundant features.

Wireless communication systems may be subject to non-Gaussian interference and impulsive noise. Therefore, the communication channel can be better modeled by heavy-tailed distributions, such as the non-Gaussian α-stable one. However, there are several promising solutions for AMR in the presence of impulsive non-Gaussian noise such as the Fractional Lower-Order Cyclic-Spectrum (FLOCS), the Spatial Sign Cyclic Correlation Function (SSCCF), the Fractional Lower-Order Cyclic Autocorrelation Function (FLOCAF), and the Cyclic Correntropy Function (CCF). The authors in [30] employed the graph-based FLOCS analysis in the α-stable impulsive noise environment for AMR. Firstly, the FLOCS of the received signal was obtained by the transformation of its fractional lower-order moments. Then, the modulation scheme was identified using the graph-based AMR mechanism by employing the extracted features from the adjacency matrices corresponding to the graph representation of FLOCS. In comparison, FLOCAF and CCF were adopted in [31], and a mathematical expression demonstrating that SSCCF is a particular case of the FLOCAF was derived. Besides, a detailed analysis of the cyclic spectrum of BPSK, QPSK, 8QAM, 16QAM, and 32QAM signals obtained by the FLOCAF and CCF was provided. Moreover, both FLOCAF and CCF were found to allow the symbol rate parameter estimation, where the latter was found to be more efficient.

#### B. ML-AMR METHODS FOR MIMO SYSTEMS

The ANN-based AMR algorithm in [20] was proposed for MIMO systems, and it was then modified in [21] by considering the time variation in the channel over an observation interval. A sliding window simplified constant modulus algorithm (SW-SCMA) was used during the blind channel estimation phase. The work reported in [21] considered a more realistic scenario because the channel matrix is unknown and blindly estimated by SCMA without any channel compensation. In [22], three AMR algorithms were presented. In the first one, the modulation schemes were directly recognized without any channel equalization. The second algorithm assumes that the channel matrix is perfectly known at the receiver side and it uses the Zero-Forcing (ZF) equalization to recover the transmitted signal symbols before features extraction. In the last one, the receiver has no prior knowledge of the channel matrix and it is blindly estimated by SCMA likewise in [20]. It is worth mentioning that the final decision was made using a majority rule by fusing all individual decisions.
In [23], two ELM algorithms were introduced for multipath fading channel without any prior knowledge of the channel and signal parameters. In spite of the good performance and the low complexity of ELM algorithm, it has some disadvantages. For instance, the number of the hidden nodes is fixed and their parameters are randomly generated to remain unaltered during the training phase. These disadvantages were handled by an algorithm called self-adaptive evolutionary ELM. The authors in [32] proposed a novel cepstrum-based pre-processing algorithm and a logarithmic functional fitting method. The main aim was to eliminate the multipath fading effects in order to improve the performance of AMR. An impartiality comparison study in [33] was introduced between four classifiers (classification tree, K-nearest neighbours, ANN, and SVM) in FB-AMR in terms of classification accuracy and computational complexity over MIMO channels. This study considered Higher Order Cumulants (HOCs) up to 6th-order, and results revealed that the ANN classifiers have the best performance/complexity trade-off for the M-ary phase-shift keying (M-PSK), M-ary Amplitude Shift Keying (M-ASK), and M-Ary Quadrature Amplitude Modulation (M-QAM) modulations.

The work presented in [34] was devoted to resolving the issue of lower recognition rate of HOC-based AMR in space division multiplexing MIMO system. Auto-Encoding Network (AEN) and ANN were used for data dimension reduction and classification, respectively. Tian et al. [35] proposed an ML approach for AMR based on a modulation-constrained clustering with unknown channel matrix and noise variance. The problem of modulation classification was converted into clustering one without direct channel estimation. However, the maximum likelihood criterion was used in order to obtain the final decision on the modulation scheme. The main core was called centroid reconstruction, in which the cluster centroids were reconstructed with fewer parameters. The model in [36] was introduced for signal detection in the Orthogonal Frequency-Division Multiplexing (OFDM) system. AEN and ELM were, respectively, exploited for features extraction and classification. Table 4 summarizes the extracted features, decision-maker, antennas number, noise nature, channel, required estimation, and the recognized modulation schemes of the previously discussed ML-AMR.

## 1) DISCUSSION AND LEARNED LESSONS

Although ML-AMR algorithms outperform the conventional methods [22], yet they still need a robust complex set of hand-crafted features to achieve a good performance. Further, these algorithms work well under relatively ideal conditions. In such methods, some pre-processing techniques have to be conducted on the received signals in order to perform some parameters estimation before AMR to enhance the classification accuracy [24]. However, ML algorithms enjoy low implementation cost with high performance on small data [26]. Nevertheless, they are time demanding and can be affected badly by the curse of dimensionality.

# IV. DEEP LEARNING IN AMR DOMAIN FOR WIRELESS COMMUNICATIONS

DL is a powerful technique that can be integrated into FB-AMR to provide a high classification accuracy along with high efficiency and robustness. It is worth noting that DNNs are deeper than simple ANNs since they consist of several layers between the input and output layers. Therefore, the more layers they have, the deeper they are. In this section, we begin by reviewing the most recent DL-AMR methods for SISO systems. Then, we discuss the existing DL models for AMR in MIMO systems. Finally, we wrap up this section with a brief discussion and learned lessons.

## A. DL-AMR METHODS FOR SISO SYSTEMS

We categorize the DL-AMR methods in this subsection based on the employed deep network models, which simplify the comparison among them. First, we outline some of the exiting signal datasets in the literature. Second, the methods that exploit CNN are discussed. Then, those using RNN are explored. Next, DNN-based AMR methods are summarized. Thereafter, we review the methods proposed for AMR based on AE models.

### 1) RELATED SIGNALS DATASETS

Signals dataset are particularly required for training, validating, and testing the networks in ML-AMR and DL-AMR models. Some researchers used their own simulated datasets, and others prefer to employ those introduced in the literature. Some of the datasets are presented as follows.

#### a: RadioML 2016.10A DATASET

In [37], a survey on the emerging applications of ML in radio signal processing domain was proposed along with a synthetic dataset. It was generated with GNU's Not Unix (GNU) Radio and it includes eight digital modulations and three analog ones. 220k signals for 20 different SNRs were generated and divided as 1k signals per modulation per SNR. In fact, it represents a cleaner and more normalized version of the RadioML 2016.04C dataset [38]. However, there is a larger version of this dataset, called RadioML 2016.10B.

#### b: RadioML 2018.01A DATASET

This dataset is one of the most challenging datasets of modulation classification which was presented in [39].<fnref n="2" /> It includes over-the-air measurements of 24 digital and analog modulation schemes spreading in a wide range of SNR values. Moreover, it contains more than 2.5M signals with synthetic simulated channel effects.

#### c: HisarMod2019.1 DATASET

In [40], 5 modulation families passing through 5 different wireless channels (ideal, static, Rayleigh, Rician, and Nakagami-m) were considered in the generated dataset. This dataset was generated using MATLAB2017. It includes 780k

<footnote n="2">²A discussion on [39] will be provided later in this paper, in section IV.</footnote>
In [23], two ELM algorithms were introduced for multipath fading channel without any prior knowledge of the channel and signal parameters. In spite of the good performance and the low complexity of ELM algorithm, it has some disadvantages. For instance, the number of the hidden nodes is fixed and their parameters are randomly generated to remain unaltered during the training phase. These disadvantages were handled by an algorithm called self-adaptive evolutionary ELM. The authors in [32] proposed a novel cepstrum-based pre-processing algorithm and a logarithmic functional fitting method. The main aim was to eliminate the multipath fading effects in order to improve the performance of AMR. An impartiality comparison study in [33] was introduced between four classifiers (classification tree, K-nearest neighbours, ANN, and SVM) in FB-AMR in terms of classification accuracy and computational complexity over MIMO channels. This study considered Higher Order Cumulants (HOCs) up to 6th-order, and results revealed that the ANN classifiers have the best performance/complexity trade-off for the M-ary phase-shift keying (M-PSK), M-ary Amplitude Shift Keying (M-ASK), and M-Ary Quadrature Amplitude Modulation (M-QAM) modulations.

The work presented in [34] was devoted to resolving the issue of lower recognition rate of HOC-based AMR in space division multiplexing MIMO system. Auto-Encoding Network (AEN) and ANN were used for data dimension reduction and classification, respectively. Tian et al. [35] proposed an ML approach for AMR based on a modulation-constrained clustering with unknown channel matrix and noise variance. The problem of modulation classification was converted into clustering one without direct channel estimation. However, the maximum likelihood criterion was used in order to obtain the final decision on the modulation scheme. The main core was called centroid reconstruction, in which the cluster centroids were reconstructed with fewer parameters. The model in [36] was introduced for signal detection in the Orthogonal Frequency-Division Multiplexing (OFDM) system. AEN and ELM were, respectively, exploited for features extraction and classification. Table 4 summarizes the extracted features, decision-maker, antennas number, noise nature, channel, required estimation, and the recognized modulation schemes of the previously discussed ML-AMR.

## 1) DISCUSSION AND LEARNED LESSONS

Although ML-AMR algorithms outperform the conventional methods [22], yet they still need a robust complex set of hand-crafted features to achieve a good performance. Further, these algorithms work well under relatively ideal conditions. In such methods, some pre-processing techniques have to be conducted on the received signals in order to perform some parameters estimation before AMR to enhance the classification accuracy [24]. However, ML algorithms enjoy low implementation cost with high performance on small data [26]. Nevertheless, they are time demanding and can be affected badly by the curse of dimensionality.

# IV. DEEP LEARNING IN AMR DOMAIN FOR WIRELESS COMMUNICATIONS

DL is a powerful technique that can be integrated into FB-AMR to provide a high classification accuracy along with high efficiency and robustness. It is worth noting that DNNs are deeper than simple ANNs since they consist of several layers between the input and output layers. Therefore, the more layers they have, the deeper they are. In this section, we begin by reviewing the most recent DL-AMR methods for SISO systems. Then, we discuss the existing DL models for AMR in MIMO systems. Finally, we wrap up this section with a brief discussion and learned lessons.

## A. DL-AMR METHODS FOR SISO SYSTEMS

We categorize the DL-AMR methods in this subsection based on the employed deep network models, which simplify the comparison among them. First, we outline some of the exiting signal datasets in the literature. Second, the methods that exploit CNN are discussed. Then, those using RNN are explored. Next, DNN-based AMR methods are summarized. Thereafter, we review the methods proposed for AMR based on AE models.

### 1) RELATED SIGNALS DATASETS

Signals dataset are particularly required for training, validating, and testing the networks in ML-AMR and DL-AMR models. Some researchers used their own simulated datasets, and others prefer to employ those introduced in the literature. Some of the datasets are presented as follows.

#### a: RadioML 2016.10A DATASET

In [37], a survey on the emerging applications of ML in radio signal processing domain was proposed along with a synthetic dataset. It was generated with GNU's Not Unix (GNU) Radio and it includes eight digital modulations and three analog ones. 220k signals for 20 different SNRs were generated and divided as 1k signals per modulation per SNR. In fact, it represents a cleaner and more normalized version of the RadioML 2016.04C dataset [38]. However, there is a larger version of this dataset, called RadioML 2016.10B.

#### b: RadioML 2018.01A DATASET

This dataset is one of the most challenging datasets of modulation classification which was presented in [39].<fnref n="2" /> It includes over-the-air measurements of 24 digital and analog modulation schemes spreading in a wide range of SNR values. Moreover, it contains more than 2.5M signals with synthetic simulated channel effects.

#### c: HisarMod2019.1 DATASET

In [40], 5 modulation families passing through 5 different wireless channels (ideal, static, Rayleigh, Rician, and Nakagami-m) were considered in the generated dataset. This dataset was generated using MATLAB2017. It includes 780k

<footnote n="2">²A discussion on [39] will be provided later in this paper, in section IV.</footnote>
<table><caption>TABLE 4. Summary of ML-AMR method for SISO and MIMO systems.</caption><thead><tr><th>Paper</th><th>Features</th><th>Classifier</th><th>Antennas number</th><th>Channel</th><th>Modulation Pool</th><th>Remarks</th></tr></thead><tbody><tr><td>[20]</td><td>HOMs up to eighth-order and HOCs up to sixth-order</td><td>ANNs</td><td>N<sub>t</sub> × N<sub>r</sub> ∈ {2×6, 3×6}</td><td>Frequency-flat time-varying MIMO</td><td>{BPSK, QPSK, 8PSK} and {BPSK, QPSK, 8PSK, 4ASK, 8ASK, 16QAM}</td><td>•Required estimation: Channel estimation.<br>•SNR range: [-10, 10] dB.<br>•Recognition accuracy with mobility: >97% for SNR>-5dB.</td></tr><tr><td>[21]</td><td>HOMs up to eighth-order and HOCs up to sixth-order</td><td>ANNs</td><td>N<sub>t</sub> × N<sub>r</sub> ∈ {2×6, 3×6}</td><td>Frequency-flat time-selective MIMO</td><td>{BPSK, QPSK, 8PSK} and {BPSK, QPSK, 8PSK, 4ASK, 8ASK, 16QAM, 64QAM}</td><td>•Required estimation: Channel estimation: Only for SCMA-DMI.<br>•SNR range: [-5, 15] dB.</td></tr><tr><td>[22]</td><td>HOMs and HOCs up to sixth-order</td><td>ANNs</td><td>N<sub>t</sub> = 2, N<sub>r</sub> = 4</td><td>Frequency-flat, spatially correlated block fading MIMO</td><td>{BPSK, QPSK, 8PSK} and {BPSK, QPSK, 8PSK, 4ASK, 8ASK, 16QAM, 64QAM}</td><td>•Required estimation: Channel estimation: Only with ZF-ELM and ZF-Sa-ELM.<br>•SNR range: [-5, 15] dB.</td></tr><tr><td>[23]</td><td>HOMs and HOCs up to sixth-order</td><td>Two ELMs</td><td>N<sub>t</sub> ∈ {2,4}, N<sub>r</sub> = 4</td><td>Time invariant and frequency flat MIMO</td><td>{BPSK, QPSK, 8PSK} and {16QAM, 64QAM}</td><td>•Required estimation: CFO and PO in the first case of known channel scenarios.<br>•SNR range: [0, 20] dB.<br>•Recognition accuracy for 2x4 antennas: ≈95% for SNR>4dB.</td></tr><tr><td>[24]</td><td>LBP histogram features</td><td>ELM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>Additive White Gaussian Noise (AWGN)</td><td>{BPSK, QPSK, 8-PSK, 16- QAM, 64-QAM and 4-ASK}</td><td>•Required estimation: CFO and PO in the first case of known channel scenarios.<br>•SNR range: [-10, 10] dB.<br>•Recognition accuracy for 2048 symbols: ≈95% for SNR>-2dB.</td></tr><tr><td>[25]</td><td>shift c, scale σ, modulation parameters f and Weight w</td><td>ELM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>AWGN and Rayleigh fading</td><td>{QPSK, 16PSK, 64PSK, BFSK, 4FSK, 16FSK, QAM,16QAM, 64QAM}</td><td>•SNR range: [-10, 5] dB.<br>•Recognition accuracy for SNR=0dB: ≈99.7% and ≈100% at 512 samples and 1024 samples, respectively.</td></tr><tr><td>[26]</td><td>Histogram features</td><td>SVM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>Multipath fading</td><td>{2ASK, QPSK and 16QAM}</td><td>•Required estimation: SNR Estimation.<br>•SNR range: [0, 30] dB.<br>•Recognition accuracy: ≈99.83% by employing all features.</td></tr><tr><td>[27]</td><td>characteristic parameters: T2, T4, T8, T16, T32</td><td>SVM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>AWGN</td><td>{2PSK, 4PSK, 8PSK and 16QAM, 32QAM and 64QAM}</td><td>•SNR range: [-2, 10] dB.<br>•Recognition accuracy: ≈59% for SNR>-2dB.</td></tr><tr><td>[29]</td><td>Higher-order cyclic cumulants (CCs)</td><td>SVM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>Multipath fading channels</td><td>{4QAM,16QAM,64QAM} and {8ASK,16QAM, 64QAM, BPSK}</td><td>•SNR range: [5, 15] dB.<br>•Recognition accuracy: >90% for SNR>10dB.</td></tr><tr><td>[19]</td><td>8 instantaneous feature, 7 HOCs, 5 wavelet features, and 5 cyclostationary features</td><td>SVM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>AWGN Channel</td><td>{2ASK, 4ASK, 8ASK, 16QAM, 2FSK, 4FSK, 8FSK, 2PSK, 4PSK, 8PSK}</td><td>•SNR range: [0, 20] dB.<br>•Recognition accuracy: ≈95% for SNR>5dB and ≈97% for SNR>10dB.</td></tr></tbody></table>

of In-Phase and Quadrature (IQ) samples spreading in the same SNR range of the aforementioned RadioML.2016.10A.

Table 5 summarizes the modulation schemes, covered SNR range, dataset's size, and the synthetic simulated channel effects of the datasets.

## 2) CNN-BASED METHODS

For a better understanding of the existing CNN-based AMR methods, they are assorted based on the adopted signal representation as follows.

### a: CLASSIFICATION USING IQ SAMPLES

IQ samples convey the changes in magnitude and phase of a wireless signal, and they are often used in RF applications. Many researchers simply used their own simulated IQ signals as the input of CNN. Shi et al. [41] conducted some pre-processing techniques and SNR estimation in order to design a CNN-based AMR method. This method aims at

eliminating the bad effects of PO in uncooperative OFDM systems and, therefore, achieving a high classification accuracy. The algorithm in [6] can learn to extract features automatically from the long symbol-rate signals but still requires high SNR to achieve high classification accuracy. A unit classifier was applied to deal with the changes in input dimensions. While two-step training method was adopted to overcome the problem of direct training, transfer learning was applied to improve the retraining efficiency. A multi-stream structure was used in [42] to increase the network width and obtain more valuable features. Several superposition convolutional units were used in each stream to overcome the problem of network training and overfitting. Moreover, a non-linear function approximator called Network-in-Network (NiN) was exploited to deal with the problem of the non-linearity in the extracted feature. Gu et al. [43] studied both blind channel identification and AMR. Two CNNs were used to remedy the flaws of
<table><caption>TABLE 4. Summary of ML-AMR method for SISO and MIMO systems.</caption><thead><tr><th>Paper</th><th>Features</th><th>Classifier</th><th>Antennas number</th><th>Channel</th><th>Modulation Pool</th><th>Remarks</th></tr></thead><tbody><tr><td>[20]</td><td>HOMs up to eighth-order and HOCs up to sixth-order</td><td>ANNs</td><td>N<sub>t</sub> × N<sub>r</sub> ∈ {2×6, 3×6}</td><td>Frequency-flat time-varying MIMO</td><td>{BPSK, QPSK, 8PSK} and {BPSK, QPSK, 8PSK, 4ASK, 8ASK, 16QAM}</td><td>•Required estimation: Channel estimation.<br>•SNR range: [-10, 10] dB.<br>•Recognition accuracy with mobility: >97% for SNR>-5dB.</td></tr><tr><td>[21]</td><td>HOMs up to eighth-order and HOCs up to sixth-order</td><td>ANNs</td><td>N<sub>t</sub> × N<sub>r</sub> ∈ {2×6, 3×6}</td><td>Frequency-flat time-selective MIMO</td><td>{BPSK, QPSK, 8PSK} and {BPSK, QPSK, 8PSK, 4ASK, 8ASK, 16QAM, 64QAM}</td><td>•Required estimation: Channel estimation: Only for SCMA-DMI.<br>•SNR range: [-5, 15] dB.</td></tr><tr><td>[22]</td><td>HOMs and HOCs up to sixth-order</td><td>ANNs</td><td>N<sub>t</sub> = 2, N<sub>r</sub> = 4</td><td>Frequency-flat, spatially correlated block fading MIMO</td><td>{BPSK, QPSK, 8PSK} and {BPSK, QPSK, 8PSK, 4ASK, 8ASK, 16QAM, 64QAM}</td><td>•Required estimation: Channel estimation: Only with ZF-ELM and ZF-Sa-ELM.<br>•SNR range: [-5, 15] dB.</td></tr><tr><td>[23]</td><td>HOMs and HOCs up to sixth-order</td><td>Two ELMs</td><td>N<sub>t</sub> ∈ {2,4}, N<sub>r</sub> = 4</td><td>Time invariant and frequency flat MIMO</td><td>{BPSK, QPSK, 8PSK} and {16QAM, 64QAM}</td><td>•Required estimation: CFO and PO in the first case of known channel scenarios.<br>•SNR range: [0, 20] dB.<br>•Recognition accuracy for 2x4 antennas: ≈95% for SNR>4dB.</td></tr><tr><td>[24]</td><td>LBP histogram features</td><td>ELM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>Additive White Gaussian Noise (AWGN)</td><td>{BPSK, QPSK, 8-PSK, 16- QAM, 64-QAM and 4-ASK}</td><td>•Required estimation: CFO and PO in the first case of known channel scenarios.<br>•SNR range: [-10, 10] dB.<br>•Recognition accuracy for 2048 symbols: ≈95% for SNR>-2dB.</td></tr><tr><td>[25]</td><td>shift c, scale σ, modulation parameters f and Weight w</td><td>ELM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>AWGN and Rayleigh fading</td><td>{QPSK, 16PSK, 64PSK, BFSK, 4FSK, 16FSK, QAM,16QAM, 64QAM}</td><td>•SNR range: [-10, 5] dB.<br>•Recognition accuracy for SNR=0dB: ≈99.7% and ≈100% at 512 samples and 1024 samples, respectively.</td></tr><tr><td>[26]</td><td>Histogram features</td><td>SVM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>Multipath fading</td><td>{2ASK, QPSK and 16QAM}</td><td>•Required estimation: SNR Estimation.<br>•SNR range: [0, 30] dB.<br>•Recognition accuracy: ≈99.83% by employing all features.</td></tr><tr><td>[27]</td><td>characteristic parameters: T2, T4, T8, T16, T32</td><td>SVM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>AWGN</td><td>{2PSK, 4PSK, 8PSK and 16QAM, 32QAM and 64QAM}</td><td>•SNR range: [-2, 10] dB.<br>•Recognition accuracy: ≈59% for SNR>-2dB.</td></tr><tr><td>[29]</td><td>Higher-order cyclic cumulants (CCs)</td><td>SVM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>Multipath fading channels</td><td>{4QAM,16QAM,64QAM} and {8ASK,16QAM, 64QAM, BPSK}</td><td>•SNR range: [5, 15] dB.<br>•Recognition accuracy: >90% for SNR>10dB.</td></tr><tr><td>[19]</td><td>8 instantaneous feature, 7 HOCs, 5 wavelet features, and 5 cyclostationary features</td><td>SVM</td><td>N<sub>t</sub> = 1, N<sub>r</sub> = 1</td><td>AWGN Channel</td><td>{2ASK, 4ASK, 8ASK, 16QAM, 2FSK, 4FSK, 8FSK, 2PSK, 4PSK, 8PSK}</td><td>•SNR range: [0, 20] dB.<br>•Recognition accuracy: ≈95% for SNR>5dB and ≈97% for SNR>10dB.</td></tr></tbody></table>

of In-Phase and Quadrature (IQ) samples spreading in the same SNR range of the aforementioned RadioML.2016.10A.

Table 5 summarizes the modulation schemes, covered SNR range, dataset's size, and the synthetic simulated channel effects of the datasets.

## 2) CNN-BASED METHODS

For a better understanding of the existing CNN-based AMR methods, they are assorted based on the adopted signal representation as follows.

### a: CLASSIFICATION USING IQ SAMPLES

IQ samples convey the changes in magnitude and phase of a wireless signal, and they are often used in RF applications. Many researchers simply used their own simulated IQ signals as the input of CNN. Shi et al. [41] conducted some pre-processing techniques and SNR estimation in order to design a CNN-based AMR method. This method aims at

eliminating the bad effects of PO in uncooperative OFDM systems and, therefore, achieving a high classification accuracy. The algorithm in [6] can learn to extract features automatically from the long symbol-rate signals but still requires high SNR to achieve high classification accuracy. A unit classifier was applied to deal with the changes in input dimensions. While two-step training method was adopted to overcome the problem of direct training, transfer learning was applied to improve the retraining efficiency. A multi-stream structure was used in [42] to increase the network width and obtain more valuable features. Several super-position convolutional units were used in each stream to overcome the problem of network training and over-fitting. Moreover, a non-linear function approximator called Network-in-Network (NiN) was exploited to deal with the problem of the non-linearity in the extracted feature. Gu et al. [43] studied both blind channel identification and AMR. Two CNNs were used to remedy the flaws of
<table><caption>TABLE 5. Summary of dataset used in the literature.</caption><thead><tr><th>Dataset</th><th>Modulation Pool</th><th>SNR range</th><th>Size</th><th>Frame length</th><th>Included effects</th></tr></thead><tbody><tr><td>[39]</td><td>Normal Classes: OOK, 4ASK, BPSK, QPSK, 8PSK, 16QAM, AM-SSB-SC, AM-DSB-SC, FM, GMSK, OQPSK<br>Difficult Classes: OOK, 4ASK, 8ASK, BPSK, QPSK, 8PSK, 16PSK, 32PSK, 16APSK, 32APSK, 64APSK, 128APSK, 16QAM, 32QAM, 64QAM, 128QAM, 256QAM, AM-SSB-WC, AM-SSB-SC, AM-DSB-WC, AM-DSB-SC, FM, GMSK, OQPSK</td><td>[-20,+30]</td><td>2.5M</td><td>1024</td><td>CFO, symbol rate offset, multipath fading, and thermal noise</td></tr><tr><td>[37]</td><td>BPSK, QPSK, 8PSK, 16QAM, 64QAM, GFSK, CPFSK, 4PAM, WB-FM, AM-SSB, and AM-DSB</td><td>[-20,+18]</td><td>220k</td><td>128</td><td>CFO, sample rate offset, AWGN, and fading</td></tr><tr><td>[40]</td><td>BPSK, QPSK, 8PSK, 16PSK, 32PSK, 64PSK, 4QAM, 8QAM, 16QAM, 32QAM, 64QAM, 128QAM, 256QAM, 4PAM, 8PAM, 16PAM, 2FSK, 4FSK, 8FSK, 16FSK, FM, PM, AM-LSB, AM-USB, AM-SC, and AM-DSB</td><td>[-20,+18]</td><td>780k</td><td>1024</td><td>multipath fading with different number of channel taps</td></tr></tbody></table>

classification on the same channel. The first CNN was responsible for identifying the channel whether it is a Line-Of-Sight (LOS) or Non-Line-Of-Sight (NLOS) channel. The second one was used to classify the modulation schemes under the identified channel. The work in [44] was devoted to making a comparison study between CNNs, RNNs, inception modules, and Convolutional Long short-term DNNs (CLDNN). Results revealed that CLDNN outperforms other networks for SNRs above -8dB.

Wang et al. [45], applied two CNNs and dropout instead of pooling operation in order to achieve high classification accuracy. On one hand, the first CNN, named DrCNN, was responsible for classifying the adopted modulation schemes (BPSK, QPSK, 8PSK, GFSK, CPFSK, PAM4) in addition to separate 16QAM and 64QAM schemes from others. On the other hand, the second CNN aimed at classifying 16QAM and 64QAM schemes. The authors created two datasets to be used in AMR. The first dataset was created by IQ samples in order to train the former CNN, while the second one adopted constellation diagrams with a density window to train the latter CNN. Simulation results demonstrated a considerable improvement when compared to the benchmark methods.

CNN and LSTM were employed in [46] for modulation scheme classification. The method in [47] was proved to have higher robustness against SNR variation and less memory consuming than the benchmark methods. Similarly, the algorithm in [48] was proposed for varying noise regimes with less computational complexity and likewise smaller model sizes. In particular, the redundant neurons were pruned through a compressive sensing based neuron pruning technology. Simulations showed that the proposed method not only reduces the computational time but also diminishes the required device memories under an acceptable performance loss. In their work, Zheng et al. [49] aimed to solve the problem of variable CNN's input size and to make full use of the complete signal burst. Thus, three fusion methods were explored: voting-based fusion, confidence-based fusion, and feature-based fusion. First, the signal is divided into multiple segments of the same length. Then, each segment is sent to the CNN to make a decision by the corresponding fusion method. Finally, all results will be fused based on classification results, confidence, or intermediate features of these segmented signals. Simulations revealed that all of the proposed fusion methods overcome the non-fusion method.

In [50], a graph mapping CNN was employed to convert the received signals into graphs. While CNN was used for feature extraction, Graph Convolutional Network (GCN) was employed for the classification part. The dataset is divided into multiple subsets for the sake of converting modulated signals into graphs. Each subset consists of 40 labeled signals and one unlabelled one.

b: CLASSIFICATION USING IQ SAMPLES FROM EXISTING DATASETS

Many researchers chose to use one of the existing IQ signals datasets<fnref n="3" /> in order to form the input of CNN. In [38], the adaptation of CNN to the dataset in [37] was studied along with a performance comparison between the proposed CNN, and those of the expert HOM-based methods. Then in [39], the authors extended their prior work on using deep CNNs for AMR [38], [44], [51]. They investigated the performance of DL-AMR considering a rigorous baseline method using HOMs and strongly boosted gradient tree classification. Besides, the effects of a wide range of design parameters, channel impairment conditions, and training dataset parameters were studied. They conducted several simulations including over-the-air measurement of AMR performance. Moreover, they built an improved version of the tool described in [37] to be used in dataset generation<fnref n="3" />.

In [52], a signal distortion correction module was proposed to improve the accuracy of CNN-based AMR. The proposed module employed an ANN in order to estimate both CFO and PO of received signals. In contrast to [41], this module eliminates the effects of both CFO and PO by shifting the signal frequency and phase prior to modulation recognition. The work reported in [53] exploited the attention mechanism for the fusion of the extracted multi-scale features. The SNR was used for optimizing the categorical cross-entropy loss with correct weight to improve the performance and the convergence time of the network. The network structure in [54] consists of six convolutional blocks (denoted M-block). Each one of these M-blocks includes three convolutional layers of different kernels. The output of every single block is the concatenation of all feature maps in the depth dimension. In order to improve the AMR accuracy, a Cyclic CNN CCNN) and a Bidirectional RNN (BRNN) were employed in [7].

<footnote n="3">³A discussion on these datasets was provided in section IV-A1.</footnote>
<table><caption>TABLE 5. Summary of dataset used in the literature.</caption><thead><tr><th>Dataset</th><th>Modulation Pool</th><th>SNR range</th><th>Size</th><th>Frame length</th><th>Included effects</th></tr></thead><tbody><tr><td>[39]</td><td>Normal Classes: OOK, 4ASK, BPSK, QPSK, 8PSK, 16QAM, AM-SSB-SC, AM-DSB-SC, FM, GMSK, OQPSK<br>Difficult Classes: OOK, 4ASK, 8ASK, BPSK, QPSK, 8PSK, 16PSK, 32PSK, 16APSK, 32APSK, 64APSK, 128APSK, 16QAM, 32QAM, 64QAM, 128QAM, 256QAM, AM-SSB-WC, AM-SSB-SC, AM-DSB-WC, AM-DSB-SC, FM, GMSK, OQPSK</td><td>[-20,+30]</td><td>2.5M</td><td>1024</td><td>CFO, symbol rate offset, multipath fading, and thermal noise</td></tr><tr><td>[37]</td><td>BPSK, QPSK, 8PSK, 16QAM, 64QAM, GFSK, CPFSK, 4PAM, WB-FM, AM-SSB, and AM-DSB</td><td>[-20,+18]</td><td>220k</td><td>128</td><td>CFO, sample rate offset, AWGN, and fading</td></tr><tr><td>[40]</td><td>BPSK, QPSK, 8PSK, 16PSK, 32PSK, 64PSK, 4QAM, 8QAM, 16QAM, 32QAM, 64QAM, 128QAM, 256QAM, 4PAM, 8PAM, 16PAM, 2FSK, 4FSK, 8FSK, 16FSK, FM, PM, AM-LSB, AM-USB, AM-SC, and AM-DSB</td><td>[-20,+18]</td><td>780k</td><td>1024</td><td>multipath fading with different number of channel taps</td></tr></tbody></table>

classification on the same channel. The first CNN was responsible for identifying the channel whether it is a Line-Of-Sight (LOS) or Non-Line-Of-Sight (NLOS) channel. The second one was used to classify the modulation schemes under the identified channel. The work in [44] was devoted to making a comparison study between CNNs, RNNs, inception modules, and Convolutional Long short-term DNNs (CLDNN). Results revealed that CLDNN outperforms other networks for SNRs above -8dB.

Wang *et al.* [45], applied two CNNs and dropout instead of pooling operation in order to achieve high classification accuracy. On one hand, the first CNN, named DrCNN, was responsible for classifying the adopted modulation schemes (BPSK, QPSK, 8PSK, GFSK, CPFSK, PAM4) in addition to separate 16QAM and 64QAM schemes from others. On the other hand, the second CNN aimed at classifying 16QAM and 64QAM schemes. The authors created two datasets to be used in AMR. The first dataset was created by IQ samples in order to train the former CNN, while the second one adopted constellation diagrams with a density window to train the latter CNN. Simulation results demonstrated a considerable improvement when compared to the benchmark methods.

CNN and LSTM were employed in [46] for modulation scheme classification. The method in [47] was proved to have higher robustness against SNR variation and less memory consuming than the benchmark methods. Similarly, the algorithm in [48] was proposed for varying noise regimes with less computational complexity and likewise smaller model sizes. In particular, the redundant neurons were pruned through a compressive sensing based neuron pruning technology. Simulations showed that the proposed method not only reduces the computational time but also diminishes the required device memories under an acceptable performance loss. In their work, Zheng *et al.* [49] aimed to solve the problem of variable CNN's input size and to make full use of the complete signal burst. Thus, three fusion methods were explored: voting-based fusion, confidence-based fusion, and feature-based fusion. First, the signal is divided into multiple segments of the same length. Then, each segment is sent to the CNN to make a decision by the corresponding fusion method. Finally, all results will be fused based on classification results, confidence, or intermediate features of these segmented signals. Simulations revealed that all of the proposed fusion methods overcome the non-fusion method.

In [50], a graph mapping CNN was employed to convert the received signals into graphs. While CNN was used for feature extraction, Graph Convolutional Network (GCN) was employed for the classification part. The dataset is divided into multiple subsets for the sake of converting modulated signals into graphs. Each subset consists of 40 labeled signals and one unlabelled one.

b: CLASSIFICATION USING IQ SAMPLES FROM EXISTING DATASETS

Many researchers chose to use one of the existing IQ signals datasets<fnref n="3" /> in order to form the input of CNN. In [38], the adaptation of CNN to the dataset in [37] was studied along with a performance comparison between the proposed CNN, and those of the expert HOM-based methods. Then in [39], the authors extended their prior work on using deep CNNs for AMR [38], [44], [51]. They investigated the performance of DL-AMR considering a rigorous baseline method using HOMs and strongly boosted gradient tree classification. Besides, the effects of a wide range of design parameters, channel impairment conditions, and training dataset parameters were studied. They conducted several simulations including over-the-air measurement of AMR performance. Moreover, they built an improved version of the tool described in [37] to be used in dataset generation<fnref n="3" />.

In [52], a signal distortion correction module was proposed to improve the accuracy of CNN-based AMR. The proposed module employed an ANN in order to estimate both CFO and PO of received signals. In contrast to [41], this module eliminates the effects of both CFO and PO by shifting the signal frequency and phase prior to modulation recognition. The work reported in [53] exploited the attention mechanism for the fusion of the extracted multi-scale features. The SNR was used for optimizing the categorical cross-entropy loss with correct weight to improve the performance and the convergence time of the network. The network structure in [54] consists of six convolutional blocks (denoted M-block). Each one of these M-blocks includes three convolutional layers of different kernels. The output of every single block is the concatenation of all feature maps in the depth dimension. In order to improve the AMR accuracy, a Cyclic CNN CCNN) and a Bidirectional RNN (BRNN) were employed in [7].

<footnote n="3">³A discussion on these datasets was provided in section IV-A1.</footnote>
The former network was devoted to extracting spatial features, while the latter meant to obtain temporal ones. Besides, the designed framework used an attention mechanism in order to improve the efficiency of the received signal features. In addition, an Auxiliary Classification Generative Adversarial Network (ACGAN) was proposed to expand the training data set. PCA procedure was used in [55] to create a set of uncorrelated features and, therefore, reduce the features vector length required for AMR.

In [40], HisarMod2019.1 dataset³ was used to evaluate the proposed CNN-based AMR method. The modulation type of each signal was firstly identified, then the signals were classified based on modulation order. The model in [56] was dedicated for beyond 5th-generation communication systems. Therefore, the main objective was to reduce the computing time below 0.01 millisecond to comply with the future communications standards. In [57], the signals were identified whether they are wideband frequency modulation or not by using two expert features: (1) the maximum value of the power spectral density of the normalized-centered instantaneous amplitude, and (2) the kurtosis of the normalized instantaneous frequency. Then, the signals were fed into a CNN-LSTM classifier where QAM16 and QAM64 schemes were considered as the same class. Finally, QAM16 and QAM64 were classified by using Haar-wavelet transform crest searching.

In [58], an adversarial transfer learning architecture was introduced using the RadioML2016.10A dataset. In order to improve the AMR overall performance, both adversarial training and knowledge transfer were exploited. The computational complexity of this architecture was found to be slightly larger than CNN and DNN. However, it was acceptable since the performance improvement was significant. Table 6 summarizes the network configuration, modulation pool, maximum achieved accuracy per SNR, and remarks of the previously discussed CNN-based AMR based on IQ samples.

### c: CLASSIFICATION USING IMAGE REPRESENTATIONS

DL technologies in the last decades have achieved outstanding achievements in the field of image processing. Therefore, many researchers in the field of wireless communication are very keen on the idea of converting the signal recognition problem into image recognition one. For example, in [68], a CNN-based AMR was presented in which various signal spectrograms were converted into an image dataset using Short-Time Fourier Transform (STFT). The authors introduced two image classification approaches in order to examine the AMR accuracy. The first one is by optimizing activation functions, while the second one is by using optimization functions. Similarly, Sun et al. [64] and Zhang et al. [69], adopted the same methodology. However, the authors in [64] implemented VGG-16 [70], a widely known CNN model, for recognizing 10 different modulation schemes. Similarly, the algorithm in [69] exploited both smooth pseudo Wigner-Ville and Born-Jordan distributions

to obtain a different kind of images. Then, these images were fused with hand-crafted features of the received signals. In the same context, images were used in [71] as the signal representation, while Wigner-Ville map pictures were used in [72]. In [73], a spectrum CNN based AMR framework and a Gaussian filter for noise elimination were presented.

The authors in [74] used constellation diagram and AlexNet CNN model [75] for network training and classification tasks. Furthermore, Caffe framework [76] was adopted as well in the whole modulation classification process. Data augmentation was applied in [77] based on ACGAN. Particularly, the data conversion algorithm converts constellation diagram to contour stellar image. Therefore, more color features were obtained compared to the constellation diagram in [74]. Moreover, this made the CNN significantly outperform the same CNN model mentioned in [74]. Traditional Generative Adversarial Nets [78] suffers from several training issues such as generator disconverge, discriminator overfitting, and mode collapse. Hence, the authors presented several measures to overcome these issues. Results showed a 0.1-6% increase in the classification accuracy using the extended dataset when comparing to the original one.

Waveform identification is considered a major part of CR technology. Therefore, Zhang et al. [79], proposed a blind method for CR waveforms recognition using Choi-Williams time-frequency Distribution (CWD). The received signals were also transformed to 2D images through CWD. Then, they were converted into a binary image by using image binarization and image denoising algorithm. In [80], an AMR method for PSK/QAM schemes of different orders was proposed based on constellation structure and k-medoids clustering in the slow and flat fading channel. Neuron pruning techniques were used in [81] to reduce not only the convolution parameters but also the number of Floating Point Operations per Second (FLOPs). This work was dedicated for AMR implementation in the edge equipment. Anyhow, the average percentage of zeros criterion was adopted for convolution layers. In addition, contour stellar image was used as the signal representation. In [82], a CNN-based AMR was proposed for the 5G signal modulation. AlexNet CNN was used for features extraction and classification. $\pi/2$-BPSK, QPSK, 16QAM, 64QAM, and 256QAM were employed as the modulation pool and the constellation was selected as the input feature of the AlexNet network. In [83], constellation diagram and transfer adaptation was adopted in features extraction subsystem, while SVM was used for classification.

Zha et al. [84], introduced a DL framework for multi-signals detection and modulation recognition. The proposed scheme can obtain the signal modulation format, center frequency, and start-stop time. Both single shot multi-box detector networks and multi-inputs CNNs were built for signal detection and modulation recognition, respectively. The time-frequency spectrum was employed as the signal characteristic expression. Hence, MFSK modulation schemes were identified during the signal detection phase while MPSK, MAPSK, and MQAM schemes were identified in
The former network was devoted to extracting spatial features, while the latter meant to obtain temporal ones. Besides, the designed framework used an attention mechanism in order to improve the efficiency of the received signal features. In addition, an Auxiliary Classification Generative Adversarial Network (ACGAN) was proposed to expand the training data set. PCA procedure was used in [55] to create a set of uncorrelated features and, therefore, reduce the features vector length required for AMR.

In [40], HisarMod2019.1 dataset³ was used to evaluate the proposed CNN-based AMR method. The modulation type of each signal was firstly identified, then the signals were classified based on modulation order. The model in [56] was dedicated for beyond 5th-generation communication systems. Therefore, the main objective was to reduce the computing time below 0.01 millisecond to comply with the future communications standards. In [57], the signals were identified whether they are wideband frequency modulation or not by using two expert features: (1) the maximum value of the power spectral density of the normalized-centered instantaneous amplitude, and (2) the kurtosis of the normalized instantaneous frequency. Then, the signals were fed into a CNN-LSTM classifier where QAM16 and QAM64 schemes were considered as the same class. Finally, QAM16 and QAM64 were classified by using Haar-wavelet transform crest searching.

In [58], an adversarial transfer learning architecture was introduced using the RadioML2016.10A dataset. In order to improve the AMR overall performance, both adversarial training and knowledge transfer were exploited. The computational complexity of this architecture was found to be slightly larger than CNN and DNN. However, it was acceptable since the performance improvement was significant. Table 6 summarizes the network configuration, modulation pool, maximum achieved accuracy per SNR, and remarks of the previously discussed CNN-based AMR based on IQ samples.

### c: CLASSIFICATION USING IMAGE REPRESENTATIONS

DL technologies in the last decades have achieved outstanding achievements in the field of image processing. Therefore, many researchers in the field of wireless communication are very keen on the idea of converting the signal recognition problem into image recognition one. For example, in [68], a CNN-based AMR was presented in which various signal spectrograms were converted into an image dataset using Short-Time Fourier Transform (STFT). The authors introduced two image classification approaches in order to examine the AMR accuracy. The first one is by optimizing activation functions, while the second one is by using optimization functions. Similarly, Sun *et al.* [64] and Zhang *et al.* [69], adopted the same methodology. However, the authors in [64] implemented VGG-16 [70], a widely known CNN model, for recognizing 10 different modulation schemes. Similarly, the algorithm in [69] exploited both smooth pseudo Wigner-Ville and Born-Jordan distributions

to obtain a different kind of images. Then, these images were fused with hand-crafted features of the received signals. In the same context, images were used in [71] as the signal representation, while Wigner-Ville map pictures were used in [72]. In [73], a spectrum CNN based AMR framework and a Gaussian filter for noise elimination were presented.

The authors in [74] used constellation diagram and AlexNet CNN model [75] for network training and classification tasks. Furthermore, Caffe framework [76] was adopted as well in the whole modulation classification process. Data augmentation was applied in [77] based on ACGAN. Particularly, the data conversion algorithm converts constellation diagram to contour stellar image. Therefore, more color features were obtained compared to the constellation diagram in [74]. Moreover, this made the CNN significantly outperform the same CNN model mentioned in [74]. Traditional Generative Adversarial Nets [78] suffers from several training issues such as generator disconverge, discriminator overfitting, and mode collapse. Hence, the authors presented several measures to overcome these issues. Results showed a 0.1-6% increase in the classification accuracy using the extended dataset when comparing to the original one.

Waveform identification is considered a major part of CR technology. Therefore, Zhang *et al.* [79], proposed a blind method for CR waveforms recognition using Choi-Williams time-frequency Distribution (CWD). The received signals were also transformed to 2D images through CWD. Then, they were converted into a binary image by using image binarization and image denoising algorithm. In [80], an AMR method for PSK/QAM schemes of different orders was proposed based on constellation structure and k-medoids clustering in the slow and flat fading channel. Neuron pruning techniques were used in [81] to reduce not only the convolution parameters but also the number of Floating Point Operations per Second (FLOPs). This work was dedicated for AMR implementation in the edge equipment. Anyhow, the average percentage of zeros criterion was adopted for convolution layers. In addition, contour stellar image was used as the signal representation. In [82], a CNN-based AMR was proposed for the 5G signal modulation. AlexNet CNN was used for features extraction and classification. $\pi/2$-BPSK, QPSK, 16QAM, 64QAM, and 256QAM were employed as the modulation pool and the constellation was selected as the input feature of the AlexNet network. In [83], constellation diagram and transfer adaptation was adopted in features extraction subsystem, while SVM was used for classification.

Zha *et al.* [84], introduced a DL framework for multi-signals detection and modulation recognition. The proposed scheme can obtain the signal modulation format, center frequency, and start-stop time. Both single shot multi-box detector networks and multi-inputs CNNs were built for signal detection and modulation recognition, respectively. The time-frequency spectrum was employed as the signal characteristic expression. Hence, MFSK modulation schemes were identified during the signal detection phase while MPSK, MAPSK, and MQAM schemes were identified in
<table><caption>TABLE 6. Summary of network configuration, modulation pool, achieved accuracy, and remarks of CNN-based AMR Methods.</caption><thead><tr><th rowspan="2">Paper</th><th colspan="5">Configuration</th><th rowspan="2">Modulation Pool</th><th rowspan="2">Accuracy per SNR</th><th rowspan="2">Remark</th></tr><tr><th>Denise</th><th>Conv</th><th>Flatten/BN</th><th>Pooling</th><th>Dropout</th><th>Training dataset size</th><th>Testing/validating dataset size</th></tr></thead><tbody><tr><td>[39]</td><td>3</td><td>7</td><td>X</td><td>7</td><td>X</td><td>(*)</td><td>(*)</td><td>RadioML 2018.01A</td><td>(**)</td><td>• Input: IQ samples. • Epochs: [0-50] • SNR Range: [-20, +20]<br>• Benchmarks: Statistical Features in XGBoost[59].<br>• Transfer learning was investigated.<br>• Refer to Table 8 for RNN configuration.<br>• (*) 80% training data and 20% for testing where dataset size∈{240k, 1M, 1.4M, 2M}.</td></tr><tr><td>[38]</td><td>2</td><td>2</td><td>X</td><td>-</td><td>✓</td><td>96k</td><td>64k</td><td>RadioML 2016.1A</td><td>>80% for SNR&gt;2dB</td><td>• Input: IQ samples. • SNR Range: [-20, +18]<br>• Benchmarks: DNN, DT, NaiveByes, KNN &amp; SVM.<br>• Epochs: 92 • Batch size: 1024</td></tr><tr><td>[41]</td><td>3</td><td>2</td><td>✓</td><td>-</td><td>✓</td><td>20k<sup>ts</sup></td><td>20k<sup>ts</sup></td><td>D1={BPSK, QPSK, 8PSK} and D2={BPSK, QPSK, 8PSK, 16QAM}</td><td>≈100% for SNR&gt;0dB (D1) and for SNR&gt;+5dB (D2).</td><td>• Input: IQ samples.<br>• Benchmarks: DT, RF, and DNN.<br>• SNR Range: [-10, +20] with 5dB step.</td></tr><tr><td>[6]</td><td>2</td><td>5</td><td>✓</td><td>4</td><td>X</td><td>79k, 228k</td><td>8.8k, 25k</td><td>{2PSK, 4PSK, 8PSK, 16QAM, 16APSK, 32APSK, 64QAM}</td><td>86% for SNR&gt;+6dB</td><td>• Input: IQ samples.<br>• Benchmarks: FB-AMR[60] &amp; ML-AMR.<br>• SNR Range: [-6, +10]</td></tr><tr><td>[42]</td><td>2</td><td>3</td><td>✓</td><td>2</td><td>X</td><td>0.5k<sup>t</sup></td><td>0.5k<sup>t</sup></td><td>{BPSK, QPSK, 8PSK, 16QAM, 64QAM, 4FSK, 8FSK, PAM, DSB, FM}</td><td>>80% for SNR&gt;2dB</td><td>• Input: IQ samples. • SNR Range: [-20, +20]<br>• Benchmarks: Resenet[61].Densenet[62], ResNeXt[63].<br>• Parameters: 590848 without 1×1 convolution kernel &amp; 153600 with it.</td></tr><tr><td>[43]</td><td>4<sup>a</sup><br>3<sup>b</sup></td><td>2</td><td>✓</td><td>-</td><td>✓</td><td>2k<sup>t</sup></td><td>2k<sup>t</sup></td><td>{2FSK, DQPSK, 16QAM, 4PAM, MSK, and GMSK}</td><td>>83% for SNR&gt;0dB, and >98% for SNR&gt;6dB</td><td>• Benchmarks: CNN,CNN without BN,RNN,HOC-DNN &amp; HOC-RF.</td></tr><tr><td>[46]</td><td>2</td><td>3</td><td>✓</td><td>2</td><td>✓</td><td>1.4k<sup>t</sup></td><td>0.6k<sup>t</sup></td><td>{BFSK, DQPSK, 16QAM, 4PAM, MSK, GMSK}</td><td>>100%, and >85% for SNR&gt;0dB under AWGN and Rayleigh, respectively.</td><td>• Input: IQ samples.<br>• Benchmarks: HOC &amp; SVM.</td></tr><tr><td>[47]</td><td>3</td><td>2</td><td>X</td><td>-</td><td>✓</td><td>6k<sup>ts</sup></td><td>6k<sup>ts</sup></td><td>{FSK, PSK, QAM}</td><td>AWGN:≈100% for SNR&gt;3dB Rayleigh:≈98% for SNR&gt;1dB</td><td>• Input: IQ samples.<br>• Benchmarks: Traditional CNN.<br>• Simulation SNR Range: [-5, +5] with an interval of 5 dB.</td></tr><tr><td>[48]</td><td>3</td><td>2</td><td>✓</td><td>-</td><td>✓</td><td>6k<sup>ts</sup></td><td>6k<sup>ts</sup></td><td>G1={BPSK, QPSK, 8PSK} and G2={BPSK, QPSK, 8PSK, 16QAM}</td><td>>80% for SNR&gt;0dB in G1 and >70% for SNR&gt;0dB in G2</td><td>• Input: IQ samples.<br>• Benchmarks: Traditional CNN-based AMR.<br>• SNR Range: [-10, +10] with 2 dB as an interval.</td></tr><tr><td>[49]</td><td>2<sup>c-</sup><br>2<sup>f</sup></td><td>2<sup>c-</sup><br>46<sup>f</sup></td><td>✓</td><td>2</td><td>✓</td><td>1k<sup>ts</sup></td><td>1k<sup>ts</sup></td><td>{BPSK, QPSK, 8PSK, OQPSK, 2FSK, 4FSK, 8FSK, 16QAM, 32QAM, 64QAM, 4PAM}</td><td>≈88% at SNR = 10dB<sup>c</sup> ≈ 94.5% at SNR = 10dB<sup>f</sup></td><td>• Input: IQ samples.<br>• Benchmarks: CNN1 and CNN2.<br>• SNR Range: [-20, +30] with an interval of 2 dB.<br>• Validation is performed every 1500 iterations.</td></tr><tr><td>[50]</td><td>1<sup>e-</sup><br>2<sup>h</sup></td><td>2<sup>e-</sup><br>4<sup>h</sup></td><td>✓</td><td>-</td><td>X</td><td>-</td><td>-</td><td>{2ASK, 4ASK, 2FSK, 4FSK, BPSK, QPSK, 16QAM and 64QAM}</td><td>≈ 69% at SNR = 0dB</td><td>• Input: IQ samples. • SNR Range: [-14, +10]<br>• Benchmarks: CNN and KNN • Parameters: 437071</td></tr><tr><td>[52]</td><td>4</td><td>4</td><td>X</td><td>2</td><td>X</td><td>-</td><td>-</td><td>RadioML 2016.01A</td><td>80% for SNR&gt;0dB</td><td>• Input: IQ samples.<br>• Benchmarks: CNN &amp; CLDNN [44].</td></tr><tr><td>[53]</td><td>25</td><td>3</td><td>X</td><td>-</td><td>X</td><td>110k</td><td>110k</td><td>RadioML2016.10A [64]</td><td>>80% for SNR&gt;2dB</td><td>• Input: IQ samples.<br>• Benchmarks: CNN, GRU, focal loss &amp; cross entropy.<br>• SNR Range: [-20, +18]</td></tr><tr><td>[54]</td><td>1</td><td>21</td><td>X</td><td>7</td><td>X</td><td>80%<br>1.6M</td><td>20%<br>0.4M</td><td>RadioML 2018.01A</td><td>85% for SNR&gt;10dB</td><td>• Input: IQ samples.<br>• Benchmarks:(ML-XGBoost,VGG,ResNet)[39], CNN-AMR[6].<br>• SNR Range: [-20, +30]<br>• Trainable parameters: 141,880</td></tr><tr><td>[55]</td><td>1</td><td>3</td><td>✓</td><td>3</td><td>X</td><td>70%<br>=<br>1.4M</td><td>30%<br>=<br>0.6M</td><td>RadioML 2018.01A</td><td>>70% for SNR&gt;7dB</td><td>• Input: IQ samples. • Epochs: 15<br>• Benchmarks: CNN[39].</td></tr><tr><td>[40]</td><td>2</td><td>4</td><td>✓</td><td>4</td><td>✓</td><td>416k<br>-<br>600k</td><td>364k<br>-<br>600k</td><td>HisarMod2019[4]<br>-<br>RML2016.10a2</td><td>>90% at SNR&gt;14dB<br>-<br>≈90% at SNR&gt;4dB</td><td>• Input: IQ samples.<br>• Benchmarks: CLDNN[65], LSTM[66].<br>• SNR Range: [-20, +18]</td></tr><tr><td>[56]</td><td>2</td><td>4</td><td>✓</td><td>2</td><td>✓</td><td>720k</td><td>480k</td><td>D1= RML2016.10b, and D2 = RML2016.10a</td><td>91.70% for SNR≈ 10dB for D1, and 83.40% at SNR = 18dB for D2</td><td>• Input: IQ samples.<br>• Benchmarks: CNN2 [51], ANN [15], CNN &amp; CNN1 [67].</td></tr><tr><td>[57]</td><td>2</td><td>3</td><td>✓</td><td>-</td><td>X</td><td>80%<br>=<br>960k</td><td>20%<br>=<br>240k</td><td>RML2016.10a</td><td>=92.3% at SNR = 10dB</td><td>• Input: IQ samples an 2 expert features.<br>• Benchmarks: CNN-LSTM[39].</td></tr><tr><td>[45]</td><td>4<sup>c</sup><br>5<sup>d</sup></td><td>2<sup>c</sup><br>3<sup>d</sup></td><td>✓</td><td>1<sup>c</sup><br>3<sup>d</sup></td><td>✓</td><td>4.2k<sup>a</sup></td><td>1.8k<sup>a</sup></td><td>{BPSK, QPSK, 8PSK, GFSK, CPFSK, 4PAM, 16QAM, and 64QAM}</td><td>95% for SNR&gt;+2dB</td><td>• Input: IQ samples in DrCNN &amp; constellation diagrams in CNN.<br>• Benchmarks: MaxCNN[51],RNN,DNN &amp; Inception.</td></tr></tbody></table>

<footnote><sup>a</sup>per type per SNR, <sup>b</sup>per type, <sup>c</sup>per SNR, <sup>d</sup>Former CNN, <sup>e</sup>Latter CNN, <sup>f</sup>DrCNN, <sup>g</sup>LatterCNN, <sup>h</sup>CNN1, <sup>i</sup>CNN2, <sup>j</sup>FECNN, <sup>k</sup>GMCNN</footnote>
<table><caption>TABLE 6. Summary of network configuration, modulation pool, achieved accuracy, and remarks of CNN-based AMR Methods.</caption><thead><tr><th rowspan="2">Paper</th><th colspan="6">Configuration</th><th rowspan="2">Modulation Pool</th><th rowspan="2">Accuracy per SNR</th><th rowspan="2">Remark</th></tr><tr><th>Denise</th><th>Conv</th><th>Flatten/BN</th><th>Pooling</th><th>Dropout</th><th>Training dataset size</th><th>Testing/validating dataset size</th></tr></thead><tbody><tr><td>[39]</td><td>3</td><td>7</td><td>X</td><td>7</td><td>X</td><td>(*)</td><td>(*)</td><td>RadioML 2018.01A</td><td>(**)</td><td>• Input: IQ samples. • Epochs: [0-50] • SNR Range: [-20, +20]<br>• Benchmarks: Statistical Features in XGBoost[59].<br>• Transfer learning was investigated.<br>• Refer to Table 8 for RNN configuration.<br>• (*) 80% training data and 20% for testing where dataset size∈{240k, 1M, 1.4M, 2M}.</td><td>• (**)>98.3%, >60%, and >67% at high SNR for the normal classes with 1M training samples, the difficult classes with 240k training samples, and with unsynchronized LOs and Doppler frequency offset with 240k training samples, respectively.</td></tr><tr><td>[38]</td><td>2</td><td>2</td><td>X</td><td>-</td><td>✓</td><td>96k</td><td>64k</td><td>RadioML 2016.1A</td><td>>80% for SNR&gt;2dB</td><td>• Input: IQ samples. • SNR Range: [-20, +18]<br>• Benchmarks: DNN, DT, NaiveByes, KNN &amp; SVM.<br>• Epochs: 92 • Batch size: 1024</td><td>• Adam solver was used.<br>• Regularization was used to prevent overfitting in CNN.<br>• Epochs take roughly 15 sec.</td></tr><tr><td>[41]</td><td>3</td><td>2</td><td>✓</td><td>-</td><td>✓</td><td>20k<sup>ts</sup></td><td>20k<sup>ts</sup></td><td>D1={BPSK, QPSK, 8PSK} and D2={BPSK, QPSK, 8PSK, 16QAM}</td><td>≈100% for SNR&gt;0dB (D1) and for SNR&gt;+5dB (D2).</td><td>• Input: IQ samples.<br>• Benchmarks: DT, RF, and DNN.<br>• SNR Range: [-10, +20] with 5dB step.</td><td>• Stochastic Gradient Descent (SGD) was used to avoid overfitting.<br>• Epochs: 500 • Batch size: 500</td></tr><tr><td>[6]</td><td>2</td><td>5</td><td>✓</td><td>4</td><td>X</td><td>79k, 228k</td><td>8.8k, 25k</td><td>{2PSK, 4PSK, 8PSK, 16QAM, 16APSK, 32APSK, 64QAM}</td><td>86% for SNR&gt;+6dB</td><td>• Input: IQ samples.<br>• Benchmarks: FB-AMR[60] &amp; ML-AMR.<br>• SNR Range: [-6, +10]</td><td>• SGD was used for minimising loss function.<br>• Adopted transfer learning and two-step training.<br>• Epochs: 20-240 • Batch size: 500-1000</td></tr><tr><td>[42]</td><td>2</td><td>3</td><td>✓</td><td>2</td><td>X</td><td>0.5k<sup>t</sup></td><td>0.5k<sup>t</sup></td><td>{BPSK, QPSK, 8PSK, 16QAM, 64QAM, 4FSK, 8FSK, PAM, DSB, FM}</td><td>>80% for SNR&gt;2dB</td><td>• Input: IQ samples. • SNR Range: [-20, +20]<br>• Benchmarks: Resenet[61].Densenet[62], ResNeXt[63].<br>• Parameters: 590848 without 1x1 convolution kernel &amp; 153600 with it.</td><td>• (NiN) multi stream network was adopted.<br>• parameter size: 164618 (R1=1, R2=1), 535754(R1=2, R2=2), 700842(R1=3, R2=3) &amp; 906890(R1=4, R2=4).<br>• Epochs: 0-55 • Batch size: 128</td></tr><tr><td>[43]</td><td>4<sup>a</sup><br>3<sup>b</sup></td><td>2</td><td>✓</td><td>-</td><td>✓</td><td>2k<sup>t</sup></td><td>2k<sup>t</sup></td><td>{2FSK, DQPSK, 16QAM, 4PAM, MSK, and GMSK}</td><td>>83% for SNR&gt;0dB, and >98% for SNR&gt;6dB</td><td>• Benchmarks: CNN,CNN without BN,RNN,HOC-DNN &amp; HOC-RF.</td><td>• SNR Range: [0, +12]<br>• Input: IQ samples. • Epochs: [0-100]</td></tr><tr><td>[46]</td><td>2</td><td>3</td><td>✓</td><td>2</td><td>✓</td><td>1.4k<sup>t</sup></td><td>0.6k<sup>t</sup></td><td>{BFSK, DQPSK, 16QAM, 4PAM, MSK, GMSK}</td><td>≈100%, and >85% for SNR&gt;0dB under AWGN and Rayleigh, respectively.</td><td>• Input: IQ samples.<br>• Benchmarks: HOC &amp; SVM.</td><td>• SNR Range: [0, +12]<br>• Refer to Table 8 for RNN configuration.</td></tr><tr><td>[47]</td><td>3</td><td>2</td><td>X</td><td>-</td><td>✓</td><td>6k<sup>ts</sup></td><td>6k<sup>ts</sup></td><td>{FSK, PSK, QAM}</td><td>AWGN:≈100% for SNR&gt;3dB<br>Rayleigh:≈98% for SNR&gt;1dB</td><td>• Input: IQ samples.<br>• Benchmarks: Traditional CNN.<br>• Simulation SNR Range: [-5, +5] with an interval of 5 dB.</td><td>• Epoch times: 40 for AWGN Channel and 55 for Rayleigh channel.<br>• Early-stopping is adopted to avoid overfitting.</td></tr><tr><td>[48]</td><td>3</td><td>2</td><td>✓</td><td>-</td><td>✓</td><td>6k<sup>ts</sup></td><td>6k<sup>ts</sup></td><td>G1={BPSK, QPSK, 8PSK} and G2={BPSK, QPSK, 8PSK, 16QAM}</td><td>>80% for SNR&gt;0dB in G1 and >70% for SNR&gt;0dB in G2</td><td>• Input: IQ samples.<br>• Benchmarks: Traditional CNN-based AMR.<br>• SNR Range: [-10, +10] with 2 dB as an interval.</td><td>• Neuron pruning via regularization was adopted.<br>• Fine-tuning and retrain the trimmed CNN in the short epochs was applied.</td></tr><tr><td>[49]</td><td>2<sup>c-</sup><br>2<sup>f</sup></td><td>2<sup>c-</sup><br>46<sup>f</sup></td><td>✓</td><td>2</td><td>✓</td><td>1k<sup>ts</sup></td><td>1k<sup>ts</sup></td><td>{BPSK, QPSK, 8PSK, OQPSK, 2FSK, 4FSK, 8FSK, 16QAM, 32QAM, 64QAM, 4PAM}</td><td>≈88% at SNR = 10dB<sup>c</sup> ≈ 94.5% at SNR = 10dB<sup>f</sup></td><td>• Input: IQ samples.<br>• Benchmarks: CNN1 and CNN2.<br>• SNR Range: [-20, +30] with an interval of 2 dB.<br>• Validation is performed every 1500 iterations.</td><td>• SGD was used to prevent overfitting.<br>• The performance of confidence-based and feature-based fusions are almost the same and they are both better than voting-based fusion.<br>• The performance gains under CNN1 is much more obvious.</td></tr><tr><td>[50]</td><td>1<sup>e-</sup><br>2<sup>h</sup></td><td>2<sup>e-</sup><br>4<sup>h</sup></td><td>✓</td><td>-</td><td>X</td><td>-</td><td>-</td><td>{2ASK, 4ASK, 2FSK, 4FSK, BPSK, QPSK, 16QAM and 64QAM}</td><td>≈ 69% at SNR = 0dB</td><td>• Input: IQ samples. • SNR Range: [-14, +10]<br>• Benchmarks: CNN and KNN • Parameters: 437071</td><td>• SGD was used • Semi-supervised training.<br>• Inference time: 0.78ms • Batch size: 10</td></tr><tr><td>[52]</td><td>4</td><td>4</td><td>X</td><td>2</td><td>X</td><td>-</td><td>-</td><td>RadioML 2016.01A</td><td>80% for SNR&gt;0dB</td><td>• Input: IQ samples.<br>• Benchmarks: CNN &amp; CLDNN [44].<br>• Input: IQ samples.<br>• Benchmarks: CNN, GRU, focal loss &amp; cross entropy.<br>• SNR Range: [-20, +18]</td><td>• SNR Range: [-20, +18]<br>• Adam optimizer was used.<br>• Adam optimizer was used.<br>• Epochs: 10<br>• BN layer &amp; Global average pooling were adopted.</td></tr><tr><td>[53]</td><td>25</td><td>3</td><td>X</td><td>-</td><td>X</td><td>110k</td><td>110k</td><td>RadioML2016.10A [64]</td><td>>80% for SNR&gt;2dB</td><td>• Input: IQ samples.<br>• Benchmarks:(ML-XGBoost,VGG,ResNet)[39], CNN-AMR[6].<br>• SNR Range: [-20, +30]<br>• Trainable parameters: 141,880</td><td>• M-block comprises 3 convolutional layers.<br>• Skip connections technique was applied.<br>• SGD was used to avoid overfitting.<br>• Accuracy increased as M-block increased.<br>• Epochs: 60 • Batch size: 128</td></tr><tr><td>[54]</td><td>1</td><td>21</td><td>X</td><td>7</td><td>X</td><td>80%<br>1.6M</td><td>20%<br>0.4M</td><td>RadioML 2018.01A</td><td>85% for SNR&gt;10dB</td><td>• Input: IQ samples. • SNR Range: [15, +18]<br>• Benchmarks: CNN[39].</td><td>• SNR Range: [-20, +30]<br>• PCA was adopted.</td></tr><tr><td>[55]</td><td>1</td><td>3</td><td>✓</td><td>3</td><td>X</td><td>70%<br>=<br>1.4M</td><td>30%<br>=<br>0.6M</td><td>RadioML 2018.01A</td><td>>70% for SNR&gt;7dB</td><td>• Input: IQ samples. • Epochs: 15<br>• Benchmarks: CNN[39].</td><td>• SNR Range: [-20, +30]<br>• PCA was adopted.</td></tr><tr><td>[40]</td><td>2</td><td>4</td><td>✓</td><td>4</td><td>✓</td><td>416k<br>-<br>600k</td><td>364k<br>-<br>600k</td><td>HisarMod2019[4]<br>-<br>RML2016.10a2</td><td>>90% at SNR&gt;14dB<br>-<br>≈90% at SNR&gt;4dB</td><td>• Input: IQ samples.<br>• Benchmarks: CLDNN[65], LSTM[66].<br>• SNR Range: [-20, +18]</td><td>• Parameters = 1576453 for HisarMod2019 and 659594 for RML2016.10a.<br>• Adam optimizer was used.</td></tr><tr><td>[56]</td><td>2</td><td>4</td><td>✓</td><td>2</td><td>✓</td><td>720k</td><td>480k</td><td>D1= RML2016.10b, and D2 = RML2016.10a</td><td>91.70% for SNR≈ 10dB for D1, and 83.40% at SNR = 18dB for D2</td><td>• Input: IQ samples.<br>• Benchmarks: CNN2 [51], ANN [15], CNN &amp; CNN1 [67].<br>• Input: IQ samples.<br>• Benchmarks: CNN- LSTM[39].</td><td>• SNR Range: [-20, +18]<br>• Gaussian noise layer was applied for regularization.<br>• Computing time = 0.0048ms<br>• Simulation SNR Range: [-20, +18]<br>• One LSTM layer was added in the network structure.</td></tr><tr><td>[57]</td><td>2</td><td>3</td><td>✓</td><td>-</td><td>X</td><td>80%<br>=<br>960k</td><td>20%<br>=<br>240k</td><td>RML2016.10a</td><td>=92.3% at SNR = 10dB</td><td>• Input: IQ samples an 2 expert features.<br>• Benchmarks: CNN-LSTM[39].</td><td>• SNR Range: [-8, +18] with an interval of 2 dB.<br>• Distinguish QAMs with high accuracy at SNR&gt;6dB.<br>• Netwrok parameters: 1,101,448</td></tr><tr><td>[45]</td><td>4<sup>c</sup><br>5<sup>d</sup></td><td>2<sup>c</sup><br>3<sup>d</sup></td><td>✓</td><td>1<sup>c</sup><br>3<sup>d</sup></td><td>✓</td><td>4.2k<sup>a</sup></td><td>1.8k<sup>a</sup></td><td>{BPSK, QPSK, 8PSK, GFSK, CPFSK, 4PAM, 16QAM, and 64QAM}</td><td>95% for SNR&gt;2dB</td><td>• Input: IQ samples in DrCNN &amp; constellation diagrams in CNN.<br>• Benchmarks: MaxCNN[51],RNN,DNN &amp; Inception.</td><td>• Netwrok parameters: 1,101,448</td></tr></tbody></table>

<footnote>a per type per SNR, b per type, c per SNR, d Former CNN, e Latter CNN, f DrCNN, g LatterCNN, h CNN1, i CNN2, j FECNN, k GMCNN</footnote>
the same categories to be then recognized using multi-inputs CNNs. In [85], a comparative analysis was provided in order to clarify the effects of choosing the features, classifiers, and datasets. Also, the effects of the presence of noises on the classification performance were investigated. Feature discrimination analysis was provided through SVM, DNN, and CNN. In [86], a two-stage hybrid digital AMR method was proposed based on STFT and CNN. Firstly, STFT was used to convert the signals into 2D images in order to extract the time-frequency features and feed the input of the CNN classifier. In [87], a CNN-based intelligent eye-diagram analyzer was proposed for modulation format recognition and optical SNR OSNR) estimation. Received signals were converted into eye diagram images that have a grid-like topology and then processed by CNN for modulation classification.

### d: CLASSIFICATION USING OTHER INPUTS

Anti-noise processing and deep sparse-filtering CNN were used in [88] to design an AMR method for very high frequencies. The Cyclic Spectra (CS) of modulated signals were calculated first. Then, Low-Rank Representation (LRR) algorithm was applied in the de-noising process of the obtained cyclic spectrum images. Zhang et al. [89] proposed a combination of the IQ samples and the 4th-order cumulants of the modulated signals. The combined vector is then fed into both CNN and LSTM classifiers, namely CNN-IQFOC and LSTM-IQFOC, respectively. In [90], the impacts of features selection for AMR were investigated. Moreover, a novel method for selecting the most effective and distinctive features from a larger set of features was proposed. This method was called Bhattacharyya distance-based feature selection algorithm. In particular, the Bhattacharyya distance metric was applied to evaluate the similarity between two different probability distributions, and calculate the highest distance for all candidate modulation schemes. This research studied the case of SISO system with perfect timing synchronization, AWGN, and frequency-selective fading channels. However, many simulations showed that the proposed algorithm achieves a substantial reduction in the computational complexity and outperforms the traditional PCA.

Teng et al. [5] extend their previous work in [91] and proposed an accumulated polar feature-based DL with a channel compensation mechanism. In [91], they have shown that learning features from polar coordinates, which can be obtained from Cartesian coordinates, can achieve higher recognition accuracy. Then in [5], they add a new temporal axis to accumulate historical information of symbols in such dimension. In the proposed method, the polar coordinates were projected to grid-like images. After that, the grid-like images were converted into colorful images to feed the CNN classifier. Moreover, an ANN-based channel estimator was presented in order to cope with the problem of fading channels. This was accomplished by finding the inverse channel response and reduce the impact of power scaling and

phase shift on the performance. Two mechanisms for online retraining were introduced to deal with the time-varying fading channel while having lower transmission and retraining overhead. The model reported in [91] was used as a benchmark. Results revealed that the proposed method can reduce the offline training overhead by about 190 times compared to [91] and, therefore, provide better efficiency and accuracy. Table 7 summarizes the network configuration, modulation pool, maximum achieved accuracy per SNR, and remarks of the previously discussed CNN-based AMR when images and other signal representations are employed.

### 3) RNN-BASED METHODS

Data augmentation techniques are considered as a proper solution for the drawbacks of insufficient training data. In [102], for example, a state-of-the-art LSTM-based AMR was presented to evaluate Gaussian noise, rotation, and flip methods of radio signals augmentation in both training and inference phases. Similarly in [103], RNN and LSTM were investigated for AMR. First, the received signals were converted into two normalized matrices to be then fed into RNN. After that, sequence-correlated features of I/Q signal components and amplitude/phase signal components were extracted using 4-layer dual-channel LSTM. In [104], a class activation vector was introduced to visualize the extracted features by using different DL-based radio modulation classifiers including CNN and LSTM classifiers. On one hand, the CNN classifier was found to be insensitive to the input formats and capture similar radio features. On the other hand, the LSTM classifier discriminates different modulation schemes in a similar manner to the knowledge of human experts. Moreover, the LeNet-based and the ResNet-based classifiers tend to capture the transitions between and around modulation reference points, respectively.

Authors in [105] outlined a Dual Path Network (DPN) for joint blind modulation classification and symbol recovery. This architecture combines both DL and linear signal processing in order to estimate signal parameters and correct signal distortions like CFO and multipath fading. It consists of a combination of residual blocks, LSTM and RNNs. Briefly, the received samples were used to generate five different outputs. Each output has a corresponding loss function different from others. Hence, the training loss was a combination of these different loss functions. Simulation results revealed that the proposed DPN not only provides good accuracy in signal distortion estimation but also outperforms many DL methods in terms of classification accuracy. In [106], a blind modulation scheme detection algorithm of non-orthogonal multiple access [107] systems was designed. ResNet [61] was used as a classifier, and joint constellation density diagrams were used as the discriminating representation. Furthermore, the wavelet de-noising method was adopted in order to reduce the bad effects of high-intensity noise and losses, thereby improve the quality of constellations.
the same categories to be then recognized using multi-inputs CNNs. In [85], a comparative analysis was provided in order to clarify the effects of choosing the features, classifiers, and datasets. Also, the effects of the presence of noises on the classification performance were investigated. Feature discrimination analysis was provided through SVM, DNN, and CNN. In [86], a two-stage hybrid digital AMR method was proposed based on STFT and CNN. Firstly, STFT was used to convert the signals into 2D images in order to extract the time-frequency features and feed the input of the CNN classifier. In [87], a CNN-based intelligent eye-diagram analyzer was proposed for modulation format recognition and optical SNR OSNR) estimation. Received signals were converted into eye diagram images that have a grid-like topology and then processed by CNN for modulation classification.

### d: CLASSIFICATION USING OTHER INPUTS

Anti-noise processing and deep sparse-filtering CNN were used in [88] to design an AMR method for very high frequencies. The Cyclic Spectra (CS) of modulated signals were calculated first. Then, Low-Rank Representation (LRR) algorithm was applied in the de-noising process of the obtained cyclic spectrum images. Zhang et al. [89] proposed a combination of the IQ samples and the 4th-order cumulants of the modulated signals. The combined vector is then fed into both CNN and LSTM classifiers, namely CNN-IQFOC and LSTM-IQFOC, respectively. In [90], the impacts of features selection for AMR were investigated. Moreover, a novel method for selecting the most effective and distinctive features from a larger set of features was proposed. This method was called Bhattacharyya distance-based feature selection algorithm. In particular, the Bhattacharyya distance metric was applied to evaluate the similarity between two different probability distributions, and calculate the highest distance for all candidate modulation schemes. This research studied the case of SISO system with perfect timing synchronization, AWGN, and frequency-selective fading channels. However, many simulations showed that the proposed algorithm achieves a substantial reduction in the computational complexity and outperforms the traditional PCA.

Teng et al. [5] extend their previous work in [91] and proposed an accumulated polar feature-based DL with a channel compensation mechanism. In [91], they have shown that learning features from polar coordinates, which can be obtained from Cartesian coordinates, can achieve higher recognition accuracy. Then in [5], they add a new temporal axis to accumulate historical information of symbols in such dimension. In the proposed method, the polar coordinates were projected to grid-like images. After that, the grid-like images were converted into colorful images to feed the CNN classifier. Moreover, an ANN-based channel estimator was presented in order to cope with the problem of fading channels. This was accomplished by finding the inverse channel response and reduce the impact of power scaling and

phase shift on the performance. Two mechanisms for online retraining were introduced to deal with the time-varying fading channel while having lower transmission and retraining overhead. The model reported in [91] was used as a benchmark. Results revealed that the proposed method can reduce the offline training overhead by about 190 times compared to [91] and, therefore, provide better efficiency and accuracy. Table 7 summarizes the network configuration, modulation pool, maximum achieved accuracy per SNR, and remarks of the previously discussed CNN-based AMR when images and other signal representations are employed.

### 3) RNN-BASED METHODS

Data augmentation techniques are considered as a proper solution for the drawbacks of insufficient training data. In [102], for example, a state-of-the-art LSTM-based AMR was presented to evaluate Gaussian noise, rotation, and flip methods of radio signals augmentation in both training and inference phases. Similarly in [103], RNN and LSTM were investigated for AMR. First, the received signals were converted into two normalized matrices to be then fed into RNN. After that, sequence-correlated features of I/Q signal components and amplitude/phase signal components were extracted using 4-layer dual-channel LSTM. In [104], a class activation vector was introduced to visualize the extracted features by using different DL-based radio modulation classifiers including CNN and LSTM classifiers. On one hand, the CNN classifier was found to be insensitive to the input formats and capture similar radio features. On the other hand, the LSTM classifier discriminates different modulation schemes in a similar manner to the knowledge of human experts. Moreover, the LeNet-based and the ResNet-based classifiers tend to capture the transitions between and around modulation reference points, respectively.

Authors in [105] outlined a Dual Path Network (DPN) for joint blind modulation classification and symbol recovery. This architecture combines both DL and linear signal processing in order to estimate signal parameters and correct signal distortions like CFO and multipath fading. It consists of a combination of residual blocks, LSTM and RNNs. Briefly, the received samples were used to generate five different outputs. Each output has a corresponding loss function different from others. Hence, the training loss was a combination of these different loss functions. Simulation results revealed that the proposed DPN not only provides good accuracy in signal distortion estimation but also outperforms many DL methods in terms of classification accuracy. In [106], a blind modulation scheme detection algorithm of non-orthogonal multiple access [107] systems was designed. ResNet [61] was used as a classifier, and joint constellation density diagrams were used as the discriminating representation. Furthermore, the wavelet de-noising method was adopted in order to reduce the bad effects of high-intensity noise and losses, thereby improve the quality of constellations.
<table><caption>TABLE 7. Summary of network configuration, modulation pool, achieved accuracy, and remarks of CNN-based AMR Methods.</caption><thead><tr><th rowspan="2">Paper</th><th colspan="5">Configuration</th><th rowspan="2">Modulation Pool</th><th rowspan="2">Accuracy per SNR</th><th rowspan="2">Remark</th></tr><tr><th>Dense</th><th>Conv</th><th>Flatten/BN</th><th>Pooling</th><th>Dropout</th><th>Training dataset size</th><th>Testing/validating dataset size</th></tr></thead><tbody><tr><td>[68]</td><td>3</td><td>2</td><td>✓</td><td>2</td><td>X</td><td>36k<sup>1</sup></td><td>16.8k<sup>1</sup></td><td>{AM, BPSK, FSK, FM} &gt;80% for SNR&gt;-6dB</td><td>• Input: STFT spectrogram images.<br>• Benchmarks: SGD[65], Adam[92] &amp; RMSProp[93].<br>• SNR Range: [-20, +20]</td><td>• Iterations &lt; 80k • Adam solver was used.<br>• The recognition accuracy under different activation functions (ReLU, Sigmoid, softmax, tanh) was studied.</td></tr><tr><td>[72]</td><td>1</td><td>2</td><td>X</td><td>2</td><td>X</td><td>4k<sup>1</sup></td><td>0.2k<sup>1</sup></td><td>{Const, LFM, SFM, PFM, and FSK}</td><td>&gt;93% when SNR&gt;0dB</td><td>• Mini-batch gradient descent was used to improve efficiency.<br>• The error rate decreases with increasing iteration number.<br>• Iterations: [0-40]</td></tr><tr><td>[73]</td><td>1</td><td>4</td><td>X</td><td>3</td><td>X</td><td>0.7k<sup>2</sup></td><td>0.3k<sup>2</sup></td><td>RadioML2016.10A [64]</td><td>&gt;80% for SNR&gt;0dB</td><td>• Input: Spectrogram images • SNR Range: [-20, +18]<br>• Benchmarks: CNNR- IQFOC [89] &amp; CNNR-IQ [38].</td><td>• Training time: 7.14 ms • Memory: 941k<br>• iterations: 15 • Epochs: 0-120 • Network parameters: 199k</td></tr><tr><td>[74], [77]</td><td>3</td><td>5</td><td>X</td><td>3</td><td>✓</td><td>10k<sup>1</sup></td><td>1k<sup>1</sup></td><td>{BPSK, 4ASK, QPSK, OQPSK, 8PSK, 16QAM, 32QAM, 64QAM} [77] &amp; {QPSK, 8PSK, 16QAM, 64QAM} [74]</td><td>&gt; 95% for SNR &gt; -2 dB [77] and for SNR&gt;8 dB [74]</td><td>• Input: contour stellar images[77] &amp; constellation diagram[74].<br>• Benchmarks: GoogleNet, SVM &amp; Cumulant.<br>• Simulation SNR Range: [-6, +14][77], [-4, +14][74].</td><td>• Data Augmentation was applied using ACGAN [77].<br>• Iterations &lt; 50k [77], 100k [74] • Batch size: 64<br>• AlexNet [75] was adopted.</td></tr><tr><td>[81]</td><td>5</td><td>2</td><td>X</td><td>-</td><td>✓</td><td>8k<sup>8</sup></td><td>8k<sup>8</sup></td><td>{4ASK, BSPK, QPSK, OQPSK, 8PSK, 16QAM, 32QAM, 64QAM}</td><td>&gt;90% for SNR&gt;-4dB</td><td>• Input: contour stellar images.<br>• Benchmarks: (AlexNet,SVM-5,AVM-7,Cumulant)[94], Orginal AlexNet.<br>• SNR Range: [-6, +6] with an interval of 2 dB.</td><td>• AlexNet was dopted with 60M parameters &amp; 727M FLOPs.<br>• This method could use only 1.5%-5% parameter and 33%-35% time without losing accuracy more than 1.2%<br>• This method was implemented on NVIDIA Jetson TX2.</td></tr><tr><td>[84]</td><td>1</td><td>3</td><td>✓</td><td>3</td><td>X</td><td>-</td><td>-</td><td>{BPSK, QPSK, OQPSK, 8PSK, 16QAM, 16APSK, 32APSK, 64QAM}</td><td>&gt;80% for SNR&gt;0dB</td><td>• Input: vector and eye diagrams.<br>• SNR Range: [0, +10]</td><td>• Benchmarks: Cumulant[95],SVM-7[4],CNNR-IQ[38] &amp; CNN[94].</td></tr><tr><td>[86]</td><td>3</td><td>5</td><td>X</td><td>3</td><td>✓</td><td>1071<sup>1</sup></td><td>459<sup>1</sup></td><td>{ASK, FSK, PSK, QASK, QFSK, QPSK}</td><td>&gt;99% for SNR&gt;0dB</td><td>• Input: Spectrograms images using STFT.<br>• Benchmarks: VGG-16[96], VGG-19[70], GoogLeNet[97], and ResNet-101[61].</td><td>• SNR Range: [0, +25]<br>• transfer learning approach was adopted.<br>• Batch size: 64 • Epoch number: 32</td></tr><tr><td>[88]</td><td>1</td><td>2</td><td>X</td><td>2</td><td>X</td><td>10k<sup>1</sup></td><td>1k<sup>1</sup></td><td>{BPSK, QPSK, 2FSK, 4FSK, MSK, AM, FM}</td><td>95% for SNR&gt;+2dB</td><td>• Input: CS-LRR samples. • SNR Range: [0, +20)<br>• Benchmarks: CS-Graph[4] &amp; RawWave-CNN[98].<br>• Fine-tune the entire CNN with supervised learning.</td><td>• Unsupervised sparse filters to pre-train the CNN.<br>• Netwrok Parameters: 3312<br>• Epochs: 5 • Batch size: 50</td></tr><tr><td>[89]</td><td>2</td><td>2</td><td>X</td><td>2</td><td>✓</td><td>0.7k<sup>2</sup></td><td>0.3k<sup>2</sup></td><td>RadioML2016.10aA [64]</td><td>&gt;80% for SNR&gt;0dB</td><td>• Input: IQFOC samples. • SNR Range: [-20, +18)<br>• Benchmarks: LSTM-IQ, CNN-IQ, and CNNR-IQ [38].</td><td>• Refer to Table 8 for LSTM configuration.</td></tr><tr><td>[90]</td><td>3</td><td>2</td><td>X</td><td>2</td><td>X</td><td>64k<sup>1</sup><br>96k<sup>2</sup></td><td>64k<sup>1</sup><br>96k<sup>2</sup></td><td>{BPSK, QPSK, 8-PSK, 16QAM}</td><td>• &gt;90% for SNR&gt;2dB for AWGN channel.<br>• &gt;80% for SNR&gt;6dB in FS-fading channels.</td><td>• Input: feature sets including HOC.<br>• Benchmarks: CNN, Sparse AE DNN (SAEDNN), Radial Basis Function Network (RBFN), and PCA. • Loss function: MSE for SAEDNN and cross-entropy for CNN.<br>• Epochs: 400<sup>1</sup> -10<sup>2</sup> • Batch size: 256<sup>2</sup></td><td>• SNR Range: [0, +15] • Iterations: 500<br>• Optimization Algo: L-BFGS for SAEDNN &amp; SGDM for CNN.<br>• The complexity can be coarsely approximated to O(NDm), where "N", "D" and "m" are - dataset size, dimension of input data and number of targets respectively.</td></tr><tr><td>[5]*</td><td>3</td><td>2</td><td>✓</td><td>2</td><td>✓</td><td>5k<sup>1</sup></td><td>1k<sup>1</sup></td><td>{QPSK, 8PSK, 16QAM, 64QAM}</td><td>(*)</td><td>• Input: Polar features. • SNR Range: [-4, +12)<br>• Benchmarks: CNN[74], CNN[91], ML[99], HLRT[100], HOC[101].<br>• Classification accuracy average: 97.9% at high SNR.<br>• Parameters: 2 / 1,032 • Time per epoch: 1.08 s</td><td>• (*) ≈ 98% for model size = 1032 and SNR = 8dB, ≈ 98% for training data size = 10k and SNR = 8dB, &gt;80% for SNR&gt;2 dB under AWGN channel, &gt;80% for SNR&gt;3 dB under fading channel.</td></tr><tr><td>[5]**</td><td>3</td><td>2</td><td>✓</td><td>2</td><td>✓</td><td>259k</td><td>317k</td><td>RadioML 2018.01A</td><td>74.2% for SNR = 20dB</td><td>• Input: IQ samples. • SNR Range: [-10, +20)<br>• Benchmarks: LSTM[66],CNN[74],ResNet[39] &amp; CNN[4].</td><td>• Time per epoch: 6s • Training time: 198s<br>• Epochs: 33</td></tr></tbody></table>

## 4) DNN-BASED METHODS

Several papers [108], [109] proposed DL-AMR where the decision on the modulation scheme is made by DNN. For instance, a novel Particle Swarm Optimization (PSO) scheme was introduced in [108] to improve the structure of DNN for optimizing the number of hidden layer nodes. Similarly, the authors in [110] employed DNN for features extraction in digital coherent receivers. Amplitude histograms of

the received signals were obtained after constant modulus algorithm equalization. Results revealed that the proposed method was non-data-aided and, therefore, it does not have any bad effects on the spectral efficiency of the system. In [111], twenty-one statistical features were extracted from the received signals in order to feed the fully connected DNN classifier. The proposed technique has a higher dimension of the feature space than the conventional AMR methods, it has

<footnote n="1">¹s per type per SNR, ²per SNR, ³per type, ⁴SAEDNN, ⁵CNN</footnote>
<table><caption>TABLE 7. Summary of network configuration, modulation pool, achieved accuracy, and remarks of CNN-based AMR Methods.</caption><thead><tr><th rowspan="2">Paper</th><th colspan="6">Configuration</th><th rowspan="2">Modulation Pool</th><th rowspan="2">Accuracy per SNR</th><th rowspan="2">Remark</th></tr><tr><th>Dense</th><th>Conv</th><th>Flatten/BN</th><th>Pooling</th><th>Dropout</th><th>Training dataset size</th><th>Testing/validating dataset size</th></tr></thead><tbody><tr><td>[68]</td><td>3</td><td>2</td><td>✓</td><td>2</td><td>X</td><td>36k<sup>1</sup></td><td>16.8k<sup>1</sup></td><td>{AM, BPSK, FSK, FM}</td><td>>80% for SNR>-6dB</td><td>• Input: STFT spectrogram images.<br>• Benchmarks: SGD[65], Adam[92] & RMSProp[93].<br>• SNR Range: [-20, +20]</td><td>• Iterations &lt; 80k • Adam solver was used.<br>• The recognition accuracy under different activation functions (ReLU, Sigmoid, softmax, tanh) was studied.</td></tr><tr><td>[72]</td><td>1</td><td>2</td><td>X</td><td>2</td><td>X</td><td>4k<sup>1</sup></td><td>0.2k<sup>1</sup></td><td>{Const, LFM, SFM, PFM, and FSK}</td><td>>93% when SNR&gt;0dB</td><td>• Input: time-frequency images using Wigner-Ville formula.<br>• Benchmarks: Methods in [4].<br>• SNR Range: [-4, +15]</td><td>• Mini-batch gradient descent was used to improve efficiency.<br>• The error rate decreases with increasing iteration number.<br>• Iterations: [0-40]</td></tr><tr><td>[73]</td><td>1</td><td>4</td><td>X</td><td>3</td><td>X</td><td>0.7k<sup>2</sup></td><td>0.3k<sup>2</sup></td><td>RadioML2016.10A [64]</td><td>>80% for SNR&gt;0dB</td><td>• Input: Spectrogram images • SNR Range: [-20, +18]<br>• Benchmarks: CNNR- IQFOC [89] &amp; CNNR-IQ [38].</td><td>• Training time: 7.14 ms • Memory: 941k<br>• iterations: 15 • Epochs: 0-120 • Network parameters: 199k</td></tr><tr><td>[74], [77]</td><td>3</td><td>5</td><td>X</td><td>3</td><td>✓</td><td>10k<sup>1</sup></td><td>1k<sup>1</sup></td><td>{BPSK, 4ASK, QPSK, OQPSK, 8PSK, 16QAM, 32QAM, 64QAM} [77] &amp; {QPSK, 8PSK, 16QAM, 64QAM} [74]</td><td>> 95% for SNR &gt; -2 dB [77] and for SNR&gt;8 dB [74]</td><td>• Input: contour stellar images[77] &amp; constellation diagram[74].<br>• Benchmarks: GoogleNet, SVM &amp; Cumulant.<br>• Simulation SNR Range: [-6, +14][77], [-4, +14][74].</td><td>• Data Augmentation was applied using ACGAN [77].<br>• Iterations &lt; 50k [77], 100k [74] • Batch size: 64<br>• AlexNet [75] was adopted.</td></tr><tr><td>[81]</td><td>5</td><td>2</td><td>X</td><td>-</td><td>✓</td><td>8k<sup>8</sup></td><td>8k<sup>8</sup></td><td>{4ASK, BSPK, QPSK, OQPSK, 8PSK, 16QAM, 32QAM, 64QAM}</td><td>>90% for SNR>-4dB</td><td>• Input: contour stellar images.<br>• Benchmarks: (AlexNet,SVM-5,AVM-7,Cumulant)[94], Orginal AlexNet.<br>• SNR Range: [-6, +6] with an interval of 2 dB.</td><td>• AlexNet was adopted with 60M parameters &amp; 727M FLOPs.<br>• This method could use only 1.5%-5% parameter and 33%-35% time without losing accuracy more than 1.2%<br>• This method was implemented on NVIDIA Jetson TX2.</td></tr><tr><td>[84]</td><td>1</td><td>3</td><td>✓</td><td>3</td><td>X</td><td>-</td><td>-</td><td>{BPSK, QPSK, OQPSK, 8PSK, 16QAM, 16APSK, 32APSK, 64QAM}</td><td>>80% for SNR&gt;0dB</td><td>• Input: vector and eye diagrams.<br>• SNR Range: [0, +10]</td><td>• AlexNet was adopted with 60M parameters &amp; 727M FLOPS.<br>• This method could use only 1.5%-5% parameter and 33%-35% time without losing accuracy more than 1.2%<br>• This method was implemented on NVIDIA Jetson TX2.</td></tr><tr><td>[86]</td><td>3</td><td>5</td><td>X</td><td>3</td><td>✓</td><td>1071<sup>1</sup></td><td>459<sup>1</sup></td><td>{ASK, FSK, PSK, QASK, QFSK, QPSK}</td><td>>99% for SNR&gt;0dB</td><td>• Input: Spectrograms images using STFT.<br>• Benchmarks: VGG-16[96], VGG-19[70], GoogLeNet[97], and ResNet-101[61].</td><td>• SNR Range: [0, +25]<br>• transfer learning approach was adopted.<br>• Batch size: 64 • Epoch number: 32</td></tr><tr><td>[88]</td><td>1</td><td>2</td><td>X</td><td>2</td><td>X</td><td>10k<sup>1</sup></td><td>1k<sup>1</sup></td><td>{BPSK, QPSK, 2FSK, 4FSK, MSK, AM, FM}</td><td>95% for SNR&gt;+2dB</td><td>• Input: CS-LRR samples. • SNR Range: [0, +20)<br>• Benchmarks: CS-Graph[4] &amp; RawWave-CNN[98].<br>• Fine-tune the entire CNN with supervised learning.</td><td>• Unsupervised sparse filters to pre-train the CNN.<br>• Netwrok Parameters: 3312<br>• Epochs: 5 • Batch size: 50</td></tr><tr><td>[89]</td><td>2</td><td>2</td><td>X</td><td>2</td><td>✓</td><td>0.7k<sup>2</sup></td><td>0.3k<sup>2</sup></td><td>RadioML2016.10A [64]</td><td>>80% for SNR&gt;0dB</td><td>• Input: IQFOC samples. • SNR Range: [-20, +18)<br>• Benchmarks: LSTM-IQ, CNN-IQ, and CNNR-IQ [38].</td><td>• Refer to Table 8 for LSTM configuration.</td></tr><tr><td>[90]</td><td>3</td><td>2</td><td>X</td><td>2</td><td>X</td><td>64k<sup>1</sup><br>96k<sup>2</sup></td><td>64k<sup>1</sup><br>96k<sup>2</sup></td><td>{BPSK, QPSK, 8-PSK, 16QAM}</td><td>>90% for SNR&gt;2dB for AWGN channel.<br>>80% for SNR&gt;6dB in FS-fading channels.</td><td>• Input: feature sets including HOC.<br>• Benchmarks: CNN, Sparse AE DNN (SAEDNN), Radial Basis Function Network (RBFN), and PCA. • Loss function: MSE for SAEDNN and cross-entropy for CNN.<br>• Epochs: 400<sup>1</sup> -10<sup>2</sup> • Batch size: 256<sup>2</sup></td><td>• SNR Range: [0, +15] • Iterations: 500<br>• Optimization Algo: L-BFGS for SAEDNN &amp; SGDM for CNN.<br>• The complexity can be coarsely approximated to O(NDm), where "N", "D" and "m" are - dataset size, dimension of input data and number of targets respectively.</td></tr><tr><td>[5]*</td><td>3</td><td>2</td><td>✓</td><td>2</td><td>✓</td><td>5k<sup>1</sup></td><td>1k<sup>1</sup></td><td>{QPSK, 8PSK, 16QAM, 64QAM}</td><td>(*)</td><td>• Input: Polar features. • SNR Range: [-4, +12)<br>• Benchmarks: CNN[74], CNN[91], ML[99], HLRT[100], HOC[101].<br>• Classification accuracy average: 97.9% at high SNR.<br>• Parameters: 2 / 1,032 • Time per epoch: 1.08 s</td><td>• (*) ≈ 98% for model size = 1032 and SNR = 8dB, ≈ 98% for training data size = 10k and SNR = 8dB, >80% for SNR&gt;2 dB under AWGN channel, >80% for SNR&gt;3 dB under fading channel.</td></tr><tr><td>[5]**</td><td>3</td><td>2</td><td>✓</td><td>2</td><td>✓</td><td>259k</td><td>317k</td><td>RadioML 2018.01A</td><td>74.2% for SNR = 20dB</td><td>• Input: IQ samples. • SNR Range: [-10, +20)<br>• Benchmarks: LSTM[66], CNN[74], ResNet[39] &amp; CNN[4].</td><td>• Time per epoch: 6s • Training time: 198s<br>• Epochs: 33</td></tr></tbody></table>

## 4) DNN-BASED METHODS

Several papers [108], [109] proposed DL-AMR where the decision on the modulation scheme is made by DNN. For instance, a novel Particle Swarm Optimization (PSO) scheme was introduced in [108] to improve the structure of DNN for optimizing the number of hidden layer nodes. Similarly, the authors in [110] employed DNN for features extraction in digital coherent receivers. Amplitude histograms of

the received signals were obtained after constant modulus algorithm equalization. Results revealed that the proposed method was non-data-aided and, therefore, it does not have any bad effects on the spectral efficiency of the system. In [111], twenty-one statistical features were extracted from the received signals in order to feed the fully connected DNN classifier. The proposed technique has a higher dimension of the feature space than the conventional AMR methods, it has

<footnote n="1">¹s per type per SNR, ²per SNR, ³per type, ⁴SAEDNN, ⁵CNN</footnote>
nonetheless a good classification performance. The extracted features were carefully selected based on the analysis of the empirical distribution for each modulation class. Simulations demonstrated good performance for both AWGN and Rician channels at different Doppler frequencies and SNRs.

In [112], a DL-AMR framework was proposed based on the IQ constellation polluted by AWGN. The noisy received samples were first normalized using the K-means algorithm. Thereafter, the signals were converted into images to be then fed into the DNN classifier. It is worth mentioning that 20% dropout was applied on hidden layers to avoid overfitting. The method in [113] consists of a large Hybrid DNN (HDNN) and a smaller Layered Resnet Network (LRN). The HDNN is composed of 1-D Resnet and LSTM-RNN layers. Cross model DL was proposed in order to reduce the consumption of both storage and time. Therefore, the low-complexity requirement can be fulfilled for real-time applications. Besides, a knowledge distillation method was also proposed to drive the learning process of LRN.

DBN classifier was exploited in [114] for AMR. First, the Spectral Correlation Function (SCF) generates 2-D images. After that, the generated images will be pre-processed and fed into the DBN classifier. To evaluate the performance, MAXNET classifier [115] and a feed-forward backpropagation ANN with a continuous output [116] were used. However, a common application of DBN is feature extraction which can be used in different concepts with different granularity. The authors of [117] propose a combination of DBN and SVM classifier. Specifically, the stacked RBM networks were used to form a DBN structure in order to extract relevant features of the received signals.

## 5) AE-BASED METHODS

Li et al. [118], introduced a Stacked Sparse AE (SSAE) and softmax regression-based DL network for AMR. They employed a cyclic spectrum in order to pre-process the received signals. Also, an unsupervised SSAE-DNN-based AMR method was proposed in [119]. Its main aim was to cope with much-neglected frequency selective fading scenarios with Doppler shift. It was particularly trained by several low complexity features such as spectral and cumulant features. Furthermore, the network was trained on a range of SNR values. Simulation results showed that the proposed method can be feasible for all channel conditions. Moreover, it can achieve a robust classification performance under several impairments such as phase-frequency impairments and Doppler shift for frequency selective fading scenarios.

Zhang et al. [120], proposed a hybrid AMR algorithm for multipath fading channels. In particular, HOCs up to sixth-order and Stacked Convolutional AEs (SCAEs) were employed. The overall process starts when the received multipath signal was down-converted to the baseband. Then, HOCs were calculated and a series of cumulant patches were generated. Subsequently, zero-phase component analysis whitening was applied to each cumulant patch to reduce

its dimension. Finally, these patches and their real modulation types were combined into pairs to feed the SCAE classifier. Dai et al. [71] proposed a novel inter-class DL-AMR method by using two SSAE for features extraction. Received signals were first converted into AF images to be then fed into SSAE. The decision on the corresponding modulation scheme was made by a softmax regression classifier. Table 8 summarizes the network configuration, modulation pool, training and testing parameters, and remarks of the previously discussed RNN, DNN and AE based AMR methods.

## B. DL-AMR METHODS FOR MIMO SYSTEMS

MIMO and massive MIMO are among the smartest antenna technologies and they are considered as key enablers in the current and future communication systems. In such systems, there are multiple antennas at both ends, the transmitter and the receiver. The increasing number of used antennas will result in a significant improvement in link range and data throughput. A survey on the emerging research on DL-based models for MIMO and massive MIMO systems can be found in the literature [121]. An investigation on the MIMO detection problem, in the presence of correlated interference over time or frequency, by employing a deep CNN was proposed in [122].

Wang et al. [123], proposed a CNN-based cooperative AMR method for the MIMO systems. In order to make the final decision on the modulation scheme, two kinds of cooperative decision rules were introduced in the decision-maker. The first kind is the voting method, while the second kind is the averaging method. In the former method, all votes on the decided modulation schemes will be gathered from each antenna, and the scheme with maximum votes will be the final decision. While in the latter method, the average of probability distribution functions achieved by all of the receiving antennas will be calculated and the scheme with the highest average probability will be the final decision. However, the decision at each antenna is equally important in the direct method, whereas each antenna has a different weight in the weighty method. Results revealed that the averaging method is better than the voting one and the weighty method is better than the direct one. Besides, they proposed a CNN/ZF-AMR method [124] for MIMO systems. ZF equalization was employed since it can increase the SNR of the received signal. The conducted simulations revealed that the performance was much better in the case of perfect Channel State Information (CSI). Furthermore, the classification performance was proved to be influenced by the imperfect CSI; and also with the number of the transmit and receive antennas. They also proposed a Transfer Learning (TL)-based semi-supervised AMR (TL-AMR) in a ZF-MIMO system [125]. The proposed method includes a novel Deep Reconstruction and Classification Network (DRCN) structure. This DRCN consists of two pipelines: Convolutional AE (CAE) and CNN. The former pipeline was used for labeled samples classification, while the former was used for the massive unlabelled samples reconstruction. Consequently, the knowledge will be transferred
nonetheless a good classification performance. The extracted features were carefully selected based on the analysis of the empirical distribution for each modulation class. Simulations demonstrated good performance for both AWGN and Rician channels at different Doppler frequencies and SNRs.

In [112], a DL-AMR framework was proposed based on the IQ constellation polluted by AWGN. The noisy received samples were first normalized using the K-means algorithm. Thereafter, the signals were converted into images to be then fed into the DNN classifier. It is worth mentioning that 20% dropout was applied on hidden layers to avoid overfitting. The method in [113] consists of a large Hybrid DNN (HDNN) and a smaller Layered Resnet Network (LRN). The HDNN is composed of 1-D Resnet and LSTM-RNN layers. Cross model DL was proposed in order to reduce the consumption of both storage and time. Therefore, the low-complexity requirement can be fulfilled for real-time applications. Besides, a knowledge distillation method was also proposed to drive the learning process of LRN.

DBN classifier was exploited in [114] for AMR. First, the Spectral Correlation Function (SCF) generates 2-D images. After that, the generated images will be pre-processed and fed into the DBN classifier. To evaluate the performance, MAXNET classifier [115] and a feed-forward backpropagation ANN with a continuous output [116] were used. However, a common application of DBN is feature extraction which can be used in different concepts with different granularity. The authors of [117] propose a combination of DBN and SVM classifier. Specifically, the stacked RBM networks were used to form a DBN structure in order to extract relevant features of the received signals.

## 5) AE-BASED METHODS

Li et al. [118], introduced a Stacked Sparse AE (SSAE) and softmax regression-based DL network for AMR. They employed a cyclic spectrum in order to pre-process the received signals. Also, an unsupervised SSAE-DNN-based AMR method was proposed in [119]. Its main aim was to cope with much-neglected frequency selective fading scenarios with Doppler shift. It was particularly trained by several low complexity features such as spectral and cumulant features. Furthermore, the network was trained on a range of SNR values. Simulation results showed that the proposed method can be feasible for all channel conditions. Moreover, it can achieve a robust classification performance under several impairments such as phase-frequency impairments and Doppler shift for frequency selective fading scenarios.

Zhang et al. [120], proposed a hybrid AMR algorithm for multipath fading channels. In particular, HOCs up to sixth-order and Stacked Convolutional AEs (SCAEs) were employed. The overall process starts when the received multipath signal was down-converted to the baseband. Then, HOCs were calculated and a series of cumulant patches were generated. Subsequently, zero-phase component analysis whitening was applied to each cumulant patch to reduce

its dimension. Finally, these patches and their real modulation types were combined into pairs to feed the SCAE classifier. Dai et al. [71] proposed a novel inter-class DL-AMR method by using two SSAE for features extraction. Received signals were first converted into AF images to be then fed into SSAE. The decision on the corresponding modulation scheme was made by a softmax regression classifier. Table 8 summarizes the network configuration, modulation pool, training and testing parameters, and remarks of the previously discussed RNN, DNN and AE based AMR methods.

## B. DL-AMR METHODS FOR MIMO SYSTEMS

MIMO and massive MIMO are among the smartest antenna technologies and they are considered as key enablers in the current and future communication systems. In such systems, there are multiple antennas at both ends, the transmitter and the receiver. The increasing number of used antennas will result in a significant improvement in link range and data throughput. A survey on the emerging research on DL-based models for MIMO and massive MIMO systems can be found in the literature [121]. An investigation on the MIMO detection problem, in the presence of correlated interference over time or frequency, by employing a deep CNN was proposed in [122].

Wang et al. [123], proposed a CNN-based cooperative AMR method for the MIMO systems. In order to make the final decision on the modulation scheme, two kinds of cooperative decision rules were introduced in the decision-maker. The first kind is the voting method, while the second kind is the averaging method. In the former method, all votes on the decided modulation schemes will be gathered from each antenna, and the scheme with maximum votes will be the final decision. While in the latter method, the average of probability distribution functions achieved by all of the receiving antennas will be calculated and the scheme with the highest average probability will be the final decision. However, the decision at each antenna is equally important in the direct method, whereas each antenna has a different weight in the weighty method. Results revealed that the averaging method is better than the voting one and the weighty method is better than the direct one. Besides, they proposed a CNN/ZF-AMR method [124] for MIMO systems. ZF equalization was employed since it can increase the SNR of the received signal. The conducted simulations revealed that the performance was much better in the case of perfect Channel State Information (CSI). Furthermore, the classification performance was proved to be influenced by the imperfect CSI; and also with the number of the transmit and receive antennas. They also proposed a Transfer Learning (TL)-based semi-supervised AMR (TL-AMR) in a ZF-MIMO system [125]. The proposed method includes a novel Deep Reconstruction and Classification Network (DRCN) structure. This DRCN consists of two pipelines: Convolutional AE (CAE) and CNN. The former pipeline was used for labeled samples classification, while the former was used for the massive unlabelled samples reconstruction. Consequently, the knowledge will be transferred
<table><caption>TABLE 8. Summary of (RNN, DNN & AE) based AMR methods.</caption><thead><tr><th>Algorithm</th><th>Network configuration</th><th>Modulation pool</th><th>Remarks</th></tr></thead><tbody><tr><td>OTA-DL-based radio signal classification [39]</td><td>Input layer, 6 residual stacks, 2 fully connected layers, and softmax output layer</td><td>RadioML 2018.01A</td><td>• ResNet performance under channel impairments &amp; versus depth was studied.<br>• Training parameters: 236,344</td></tr><tr><td>CNN-RNN-Based AMR [46]</td><td>Input layer, 3 LSTM layers, 3 fully connected layers and softmax output layer</td><td>{BFSK, DQPSK, 16QAM, 4PAM, MSK and GMSK}</td><td>Refer to Table 7</td></tr><tr><td>SSAE-AF [71]</td><td>Input layer, hidden layer and output layer</td><td>{ASK, PSK, QAM, FSK, MSK, LFM, and OFDM}</td><td>• Training data: (6*7)k • Testing data: (1*7)k<br>• Batch size: 100 • Epochs: 100<br>• LSTM-IQFOC performance was around higher 2% than CNN-IQFOC (Table 7) for SNR&gt;-.5dB.</td></tr><tr><td>LSTM-IQFOC [89]</td><td>Input layer, 3 convolution layers, 1 LSTM layer, 1 fully connected layer and softmax output layer</td><td>Refer to Table 7</td><td>• Training data: 110-440k • Testing data: 110k<br>• Batch size: 128 • Epochs: 80<br>• Training data: 110k • Testing data: 110k<br>• Batch size: 256 • Epochs: 70<br>• Training data: 88k • Testing Data: 11k<br>• Batch size: 128 • Epochs: 150<br>• Validation Data: 11k • Adam optimizer was employed.</td></tr><tr><td>LSTM-based AMR [102]</td><td>Input layer, 2 LSTM layers, one fully connected layer and softmax output layer</td><td>RadioML2016.10a</td><td>• Tanh &amp; RELU activation functions were used.</td></tr><tr><td>DC-LSTM AMR [103]</td><td>Input layer, 2 dual-channel LSTM layers and softmax output layer</td><td>RadioML2016.10a</td><td>• Combine both rotation and flip methods can provide further improvement.</td></tr><tr><td>DL-based Radio Modulation Classification [104]</td><td>Input layer, 2 LSTM layers, 2 fully connected layers and softmax output layer</td><td>RadioML2016.10a</td><td>• Center loss and weibull distribution.</td></tr><tr><td>PSO-DNN [108]</td><td>Input layer, 2 hidden layers and softmax output layer</td><td>{BPSK, QPSK, 8PSK} and {16QAM, 64QAM, 256QAM}</td><td>• No. PS: (10-20-30-50)<br>• Iteration number of training/optimizing: 300/5<br>• SGD method used for training the hidden layers and MSE function to calculate the output error.</td></tr><tr><td>DNN [109]</td><td>Input layer, 2 hidden layers and softmax output layer</td><td>{2FSK, 4FSK, 2PSK, 4PSK, 2ASK, 4ASK}</td><td>• Epochs: 278<br>• Loss function: cross-entropy</td></tr><tr><td>DNN-based MFI [110]</td><td>Input layer, 2 hidden layers and softmax output layer</td><td>{QPSK, 16QAM, 64QAM}</td><td>• Training data: 135*3 • Testing data: 60*3<br>• Unsupervised pre-training and supervised fine-tuning.</td></tr><tr><td>DNN-based AMR [111]</td><td>Input layer, 3 hidden layers and softmax output layer</td><td>{BPSK, QPSK, 8PSK, 16QAM, 64QAM}</td><td>• Batch size: 50 • Epochs: 15k<br>• Training data: (15*5)k • Validation Data: (5*5)k<br>• Cost function: negative-log- likelihood.</td></tr><tr><td>DNNG [112]</td><td>Input layer, 3 hidden layers and softmax output layer [48]</td><td>{16QAM, 4QAM, 8PSK, 16PSK, BPSK, 8QAM, 32QAM}</td><td>• The best performance results was found to be at epoch equal to 2M.</td></tr><tr><td>DBN-SCF-based AMR [114]</td><td>5 hidden layers and an output layer</td><td>{4FSK, 16QAM, BPSK, QPSK, and OFDM}</td><td>• Training data: 6k • Testing data: 500<br>• Verification data: 2k • Semi-supervised learning</td></tr><tr><td>SSAE [118]</td><td>Input layer, hidden layer and output layer</td><td>{FSK, PSK, ASK, QAM and MSK}</td><td>• Training data: (2.5*5)k • Testing data: (1*5)k<br>• Batch size: 100 • Epochs: 100<br>• Unsupervised learning.</td></tr><tr><td>SCAE-HOC [120]</td><td>Input layer, six hidden layers (3 convolutional layers, 2 pooling layers and one fully connected layer)</td><td>{BPSK, QPSK, 16-QAM, and 64-QAM}</td><td>• Training data: 120k • Testing data: 60k<br>• Unsupervised pre-training and fine-tuning were performed.<br>• SGD was performed to reduce the training error.</td></tr></tbody></table>

from CAE to the feature layer of CNN in order to cope with ineffective training and over-fitting, which is caused by using limited labeled samples. However, it is worth mentioning that the modulation schemes used in all of these researches were PSK, QPSK, 8PSK, and 16QAM. Besides, the CNN structure was also the same including 2 convolutional layers, 3 dense layers, and a softmax output layer. Furthermore, the CAE in [125] had the same structure as in CNN with three deconvolution layers.

## 1) DISCUSSION AND LEARNED LESSONS

DL models play an important role in AMR and the performance demonstrated by recent prototypes is remarkable. This is because DL can prove advantageous with regard to feature learning, performance, complexity, and generalization capabilities [2]. On one hand, training DL models on a wide range of SNR can eliminate the requirement of SNR estimation and generalize the classification process in the different environmental conditions [39]. However, maintaining low computational complexity, in such models, remains challenging as well as constructing and gathering a large dataset of signals in order to train the model and to guarantee the performance

<table><caption>TABLE 9. Summary of DL-AMR method for MIMO systems.</caption><thead><tr><th>Paper</th><th>Training &amp; Testing parameters</th><th>Remarks</th></tr></thead><tbody><tr><td>[123]</td><td>• Training data: 14k<sup>18</sup><br>• Validation data: 6k<sup>18</sup><br>• Testing data: 10k<sup>18</sup></td><td>• ReLU &amp; dropout were used.<br>• Antennas number: N<sub>r</sub> × N<sub>t</sub> ∈ {4×1, 4×2, 4×4}.<br>• Benchmark: HOC and ANN-based AMR.<br>• Classification accuracy: ≈100%, >91%, >87% for SNR&gt;-.0dB when N<sub>r</sub> × N<sub>t</sub> ∈ {4×1,4×2,4×4}, respectively.</td></tr><tr><td>[124]</td><td>• Training data: 20k<sup>18</sup><br>• Testing data: 10k<sup>18</sup><br>• Maximum epoch, early-stopping epoch, batch size were set as 100, 20 and 500, respectively</td><td>• ReLU and Dropout were used.<br>• Antennas number: N<sub>r</sub> × N<sub>t</sub> ∈ {4×1, 4×2, 4×4}.<br>• Benchmark: HOC and ANN-based AMR.<br>• Classification accuracy: >90%, >83%, >81% for SNR&gt;-.5dB when N<sub>r</sub> × N<sub>t</sub> ∈ {4×1,4×2,4×4}, respectively.</td></tr><tr><td>[125]</td><td>• Training data: 20k<sup>18</sup><br>• Testing data: 10k<sup>18</sup><br>• Maximum epoch, and batch size were set as 500, and 500, respectively.</td><td>• ReLU used for all layers, while softmax and PReLU were used for the output layer in CNN and the last deconvolution layer in CAE.<br>• Dropout were used.<br>• Antennas number: N<sub>r</sub> × N<sub>t</sub> ∈ {4×1, 4×2, 4×4}.<br>• Benchmark: CNN/ZF-AMR [125].<br>• Classification accuracy: >90%, >82%, and >80% for SNR&gt;-.5dB when N<sub>r</sub> × N<sub>t</sub> ∈ {4×1,4×2,4×4}, respectively.</td></tr></tbody></table>

<sup>18</sup>per type per SNR

under varying noise regimes. For example, the CNN model reported in [42] enjoys a high classification accuracy but suffers from a large number of training parameters and, hence,