## 1 Introduction  
Fifth-generation (5G) mobile systems are being developed worldwide with the objectives of increasing system capacity and data rates, achieving low latency, improving user throughput, and supporting \"Internet of Things\" services. These systems are expected to deliver peak data rates of up to 20 Gbps and 100 times higher typical data rates per user (user throughput) [1; 2; 3; 4]. Fourth-generation (4G) mobile systems such as long term evolution (LTE) and LTE-Advanced mobile systems apply quadrature phase shift keying (QPSK), 16-quadrature amplitude modulation (QAM), and 64-QAM for the symbol modulation of orthogonal frequency division multiplexing (OFDM), as defined by Release 8 of the Third Generation Partnership Project (3 GPP) standards body. Although the highest modulation scheme under Release 8 was 64-QAM for LTE, 3 GPP Release 12 discussed the implementation of 256-QAM, and more recently, 3 GPP Release 15 introduced 1024-QAM support. Nevertheless, the application areas and/or conditions for these post-64-QAM schemes are currently very limited [5; 6; 7]. One of the most noteworthy technological pathways toward implementing 5G is increasing infrastructure density, which can reduce the distance between the user equipment (UE) and evolved node B (eNB), thereby improving the link budget between UE and eNB. The small-cell strategy is one of a number of denser infrastructure approaches in which a macro-cell is divided into several small-cells [8; 9], which can improve both system capacity and link budget because each eNB in a small-cell is directly connected to a core network. As another small-cell strategy, the concept of a phantom cell, has been proposed to provide efficient management of radio resources and mobility operation, where small-cells are overlaid on a macro-cell [10; 11]. This approach splits the control and user-data planes; that is, the macro-eNB manages all UE within the overlaid cells, although UE can connect to a neighboring low-power eNB in the small-cell. Heterogeneous networks (HetNets) are another solution of denser infrastructures in which small- or pico-cells with low-power eNBs are installed within the coverage area of a macro-cell [12; 13; 14; 15; 16]. The objective of HetNet is to allow the UE to access pico-cells that overlap within a geographical coverage area even when the UE is within the coverage of a donor macro-cell. In other words, when the traffic in the donor macro-cell increases, its UE can access the pico-cells automatically. Through this mechanism, HetNet can increase system capacity, particularly when there is the heavy traffic in the macro-cell. Relay systems are an alternative approach to denser infrastructures that are primarily useful in improving cell edge performance in macro-cells [17; 18; 19]. In this scheme, a relay node (RN) installed at a macro-cell edge where the received signal power from the serving eNB is low can transmit a strengthened signal to the UE. Thus, the RN can physically reduce the distance between eNB and UE. Three-dimensional (3D) beamforming is another key technology for achieving 5G in which antenna beams can be individually tailored to each UE in the elevation domain through the use of a massive multiple-input and multiple-output (MIMO) antenna architectures. The use of 3D beamforming by MIMO antennas enables directs signal transmission toward a target UE, potentially increasing the received signal-to-noise ratio (SNR) or signal-to-interference plus noise ratio (SINR) while reducing the interference directed to adjacent cells and other UEs [20; 21; 22]. New technologies such as the small-cell strategy, HetNets, relay systems, and 3D beamforming that improve the received SINR will enhance the introduction of higher-order QAM schemes. Previously, we proposed OFDM-based 256-, 1024-, and 4096-QAM for use in mobile systems and determined the fundamental transmission performance needed for the use of hard-decision Viterbi decoding in a typical additive white Gaussian noise (AWGN) channel [23; 24; 25; 26].