## 1 Introduction  
High-order modulations are important to improve the data rate and spectral efficiency of satellite communication systems [1]. However, the non-linear distortion due to the non-linear transfer characteristic of the high power amplifier (HPA) would reduce the overall performance gain expected with the application of higher modulation orders [2]. Therefore, the appropriate high-order constellation which is less influenced by non-linear effects is expected for high throughput satellite communications. High-order modulations commonly used are _M_-ary phase-shift keying (_M_-PSK), _M_-ary quadrature amplitude modulation (_M_-QAM) and M-ary amplitude and phase-shift keying (_M_-APSK). Generally, _M_-PSK with constant envelope is robust against the non-linear distortion, but its anti-noise performance is worse than that of _M_-QAM if the modulation order is larger than four. However, high-order QAM usually has high peak-to-average power ratio (PAPR), and it is not suitable for the satellite communication system due to its high sensitivity to the non-linear distortion [3, 4]. APSK has good performance in both anti-noise and robustness to non-linear distortion when the modulation order is greater than eight, and proper constellations suitable for the satellite communication system could be achieved through the optimization design [5, 6, 7, 8]. The second generation digital video broadcasting over satellite (DVB-S2) standard recommends 8-PSK, 16-APSK, and 32-APSK [9, 10]. Moreover, the scheme of combining the high-order modulation with error correction codes could guarantee the reliability of data transmissions and has been widely applied to the satellite communication system. In addition, the combination of 8-PSK and low-density parity-check (LDPC) codes is a very popular candidate scheme for satellite communication system because of the robustness of 8-PSK against the non-linear distortion and the excellent error correction performance of the LDPC codes [11]. However, compared to other constellations, such as 8-QAM, 5-QAM, and 9-QAM presented in [12; 13] and the general hexagonal constellation [14; 15], 8-PSK only has a ring and wastes the space of little amplitude, which results in the poor anti-noise performance and the limitation for the error performance of this scheme. In [16], 5-QAM is used as the modulation method in the solution for the fifth-generation (5G) cellular networks. For a better trade-off between the power efficiency and modulation order, we propose a new 7-point constellation based on the hexagonal grid signal sets which is optimum in the two-dimensional (I-Q) space [14; 15; 17]. The new 7-QAM constellation is also with a point in the original point as 5-QAM and 9-QAM, and it is expected to achieve better error performance over the non-linear satellite channel than the conventional 8-PSK and 8-QAM because of its excellent natures of both PAPR and minimum Euclidean distance [18; 19; 20], which means that it has better anti-noise performance and simultaneously is less influenced by the non-linear effects. Moreover, for the reliability of data transmission, the new 7-QAM constellation is combined with non-binary LDPC (NB-LDPC) codes over GF(7) because its modulation order is not the power of 2 and does not match the binary codes. In addition, the NB-LDPC codes, often with better error correction performance than binary LDPC codes [21], could be better adapted to high-order modulations without considering the inter-conversion between bit probability and symbol probability. These advantages of the 7-QAM constellation are verified through the calculating of PAPR, the derivation of an exact intuitive geometric infinite double series for its symbol error probability (SEP) over the Additive White Gaussian Noise (AWGN) channel using the similar derivation in [22; 23; 24; 25; 26; 27; 28; 29; 30] and the analysis of its sensitivity to the nonlinearity of HPAs. Finally, the demodulation threshold of 7-QAM and the symbol error rate (SER) performance of the proposed coded modulation scheme are simulated.