POSTPRINT (ACCEPTED VERSION), 10.1109/LSP.2018.2882696

1

Performance Analysis and Constellation Optimization ofStar-QAM-Aided Differential Faster-than-Nyquist Signaling
Chie Sagayama, Student Member, IEEE, Takumi Ishihara, Student Member, IEEE, andShinya Sugiura, Senior Member, IEEE
Abstract—In this letter, motivated by the recent differential faster-than-Nyquist (DFTN) signaling concept, we propose
an improved 16-point double-ring star quadrature amplitude
modulation (QAM)-aided DFTN signaling transmission, whichallows us to attain a higher bandwidth efficiency as well as asimple receiver based on noncoherent detection. We derive ananalytical error-rate bound for the proposed star-QAM DFTN
signaling in an uncoded scenario. The derived bound is used for
optimizing the star-QAM constellation for our DFTN signaling
in terms of error-rate performance in an uncoded scenario.
Our simulation results demonstrate that the proposed star-QAM
DFTN scheme outperforms its conventional phase-shift-keyingbased DFTN counterpart.

I. I NTRODUCTION
HE concept of faster-than-Nyquist (FTN) signaling was
discovered in the 1970s [1, 2] and acts as a means ofincreasing a transmission rate without expanding the bandwidth. The symbol interval of FTN signaling T is set toa value less than the first Nyquist criterion T0 . One of themain limitations imposed on FTN signaling is that the receiver
suffers from inter-symbol interference (ISI), which is specificto FTN signaling. In order to combat this limitation, several
computationally efficient frequency-domain equalizers havebeen developed [3–6], all of which rely on cyclic prefix(CP)-assisted frequency-domain equalization (FDE), which
achieve a practically low detection complexity even in a highly
dispersive frequency-selective channel, in addition to FTNinduced ISI.
In order to reduce the pilot overhead associated with channel
estimation at the receiver, an FTN pilot (FTNP) sequence
was employed in [6, 7], where efficient FTNP-based channel
estimation (CE) algorithms were developed in the frequency
domain. However, the benefit of such reduced pilot overhead isachieved at the cost of additional complexity, which is imposedby the iterative process at the receiver. More specifically,
the FTNP-based CE schemes [6, 7] still rely on the useof pilot symbols. Most recently, in order to dispense with
CE at the receiver, differential FTN (DFTN) signaling was
proposed in [8], where differential phase-shift keying (DPSK)
is employed in the context of the conventional FTN signaling.
Although in general, noncoherent detection is possible only
for an ISI-free non-dispersive channel, noncoherent detectionT

c 2018 IEEE. Personal use of this material is permitted. Permission from⃝IEEE must be obtained for all other uses, in any current or future media,
including reprinting/republishing this material for advertising or promotional
purposes, creating new collective works, for resale or redistribution to servers
or lists, or reuse of any copyrighted component of this work in other works.
C. Sagayama and T. Ishihara are with the Department of Computerand Information Sciences, Tokyo University of Agriculture and Technology,
Kogani, Tokyo 184-8588, Japan. S. Sugiura is with the Institute of Industrial
Science, University of Tokyo, Meguro-ku, Tokyo 153-8505, Japan (e-mail:sugiura@ieee.org). (Corresponding author: Shinya Sugiura.)of ISI-contaminated DFTN signaling becomes realistic in [8],by exploiting the fact that an FTN-specific ISI matrix is known
in advance of transmissions at the receiver.
The conventional differential star-quadrature amplitude
modulation (QAM) scheme typically exhibits a better performance than the DPSK scheme [9, 10], and hence a similar
benefit is also expected in the DFTN signaling scenario.
However, it is an open issue to optimize a constellation of starQAM in DFTN signaling, unlike in the conventional Nyquistcriterion-based scenario [9, 10]. This is because the optimum
star-QAM constellation in DFTN signaling may depend on the
bit-energy-to-noise ratio (Eb /N0 ) as well as the DFTN-specificsymbol packing ratio α = T /T0 .
Against the above-mentioned backcloth, the novel contributions of this letter are as follows. We first propose an improved
DFTN signaling based on 16-point star-QAM in order toenhance the bandwidth efficiency of the conventional BPSKmodulated DFTN signaling scheme [8] while maintaining the
DFTN scheme’s fundamental benefits. We derive the analyticalerror-rate bound of the proposed 16-point star-QAM-aided
DFTN signaling scheme in an uncoded scenario. The derived
bound is exploited for optimizing the 16-point star-QAM constellation in the proposed scheme. Our analytical and numerical results demonstrate that the above-mentioned fundamental
benefits of the proposed star-QAM-aided DFTN signaling areachievable. Furthermore, it is found that our scheme exhibits
an explicit advantage over the existing coherent counterpart,
especially in a rapidly time-varying channel.
II. S YSTEM M ODEL OF S TAR -QAM-A IDED DFTN
In this section, we present the system models of the conventional Nyquist-criterion star-QAM-aided differential modulation and our star-QAM-aided DFTN signaling. In this letter,
we consider a double-ring star-QAM having a constellation
size of 16, similar to [11]. The amplitudes of the inner andouter rings are denoted by aL and aH , and γ = aH /aL isreferred to as the ring ratio.
A. The Conventional Star-QAM-Aided Differential Modulation [10]We first review the conventional star-QAM-aided differential modulation. At the transmitter, a 4N -length bit sequenceb = [b1 , · · · , b4N ]T ∈ Z4N per block is differentially modulated onto 16-star-QAM symbols s = [s0 , · · · , sN ]T ∈ CN +1
as follows:
sk = ak vk

for

1 ≤ k ≤ N,

(1)

where ak ∈ {aH , aL } is the amplitude of sk , which ismodulated by b4k . For example, we have ak = aL for b4k = 0,
and otherwise we have ak = aH .

POSTPRINT (ACCEPTED VERSION), 10.1109/LSP.2018.2882696

2

Furthermore, vk (|vk | = 1) corresponds to a differentiallymodulated phase of sk , which is given by
vk = wk vk−1 ,(2)and wk is an 8-PSK symbol, which is modulated basedon three bits, i.e., [b4k−3 , b4k−2 , b4k−1 ]. Note that the initial
amplitude and 8-PSK symbol are set to a0 = aL and v0 = 1,
respectively.
The differentially encoded star-QAM signals are transmitted
to the receiver over a frequency-flat Rayleigh fading channel.
The received symbols yk ∈ C (k = 0, · · · , N ) is expressed asfollows:
yk = hk sk + nk = hk ak vk + nk ,(3)

where hk ∈ C is the channel coefficient, and nk ∈ C is theassociated additive white Gaussian noise (AWGN) component,
which obeys a complex-valued Gaussian distribution having azero mean and a noise variance of N0 . By assuming that twoconsecutive channel coefficients remain constant, i.e., hk =hk−1 , the received signals yk of (3) are rewritten as
yk = hk−1 ak vk−1 wk + nk =akyk−1 wk + ñk ,
ak−1

(4)

where we have an equivalent noise component of ñk =−(ak /ak−1 )nk−1 wk + nk , which obeys a complex-valued
Gaussian distribution having a zero mean and a variance of{(ak /ak−1 )2 + 1}N0 . Note that ak /ak−1 of (4) has value 1,γ, or 1/γ.
Having obtained the modified received signal model of (4),four information bits [b4k−3 , b4k−2 , b4k−1 , b4k ] are estimatedby demodulating (ak , wk ) without channel estimation, basedon the maximum likelihood (ML) criterion, as follows: [12](âk , ŵk ) =arg max(ak ,wk ){}∗)|ak wk∗ + âk−1 ŵk−1
|2 (
2
2
−
|w
|
+
|
ŵ
|
,
k
k−1
|ak |2 + |âk−1 |2(5)where âk and ŵk are the estimated amplitude and phase, ak
is aL or aH , and w is an 8-PSK symbol and the equation
holds when Eb /N0 is fully large [12].

Note that in this model, the Eb /N0 of the DFTN signalingis the same as that of the conventional Nyquist-criterionbased counterpart. By assuming a quasi-static frequency-flatRayleigh fading channel, the kth sampled symbol zk at thereceiver is given by∑s̃n g((k − n)T ) + η(kT ),(7)
zk = hn∫
where
we have g(t) =q(τ )q ∗ (τ − t)dτ and η(t) =∫∗n(t)q (τ − t)dτ . Additionally, η(kT ) (k = 0, · · · , N + 2ν)are the colored noise components, which have the relationshipE[η(kT )η(jT )] = N0 g((k − j)T ). Here, E[·] denotes the
expectation operation.
Furthermore, by removing both the ν head and the last νsymbols from the N + 2ν + 1 sampled symbol block, thereceived block is obtained as follows:ŷ = hGs + η,

where G ∈ R
is a circulant matrix that corresponds to the FTN-specific ISI effects. Moreover, G isexpressed by G = QT ΛQ∗ with the aid of the eigenvalue decomposition, where Q is a normalized discrete Fourier transform (DFT)√ matrix whose kth row and lth column componentis (1/ N + 1) exp [−2πj(l − 1)(k − 1)/(N + 1)]. Hence,by carrying out an inverse DFT operation on (4), we arriveat the frequency-domain received signals ofŷf = Q∗ ŷ= hΛQ∗ s + Q∗ η.

ẑ = [ẑ0 , · · · , ẑN ]T ∈ CN +1= QT Wŷf ,where we have(

B. The Proposed Star-QAM-Aided DFTN Signalingn(9)

Since the FTN-specific ISI matrix G is calculated by thesymbol packing ratio α and the roll-off factor of an RRC filter
β, G is accurately available at the receiver, versus requiring
estimation.
With the aid of the approximated noise-whitening minimum
mean-square error (MMSE) criterion [6], the estimates of z =hs are calculated as follows:

W = ΛHIn the proposed transmitter, similar to the conventional
differential 16-star-QAM in Section II-A, 4N information bits
are modulated onto (N + 1)-length differential 16-star-QAM
symbols s. Then, a 2ν-length CP is added to the head of s, inorder to obtain (N + 2ν + 1)-length symbols s̃ ∈ CN +2ν+1 .
Note that ν is assumed to be sufficiently longer than theeffective tap length induced by DFTN signaling. Here, theeffective tap length includes both the ISI effects, induced by
the FTN signaling and the frequency-flat channels, similar to[13]. Then, the transmitted signals s̃(t) are generated by bandlimiting s̃ with the aid of an root raised cosine (RRC) filterq(t) having a roll-off factor of β, as follows:∑(6)s̃n q(t − nT ).
s̃(t) =

(8)

(N +1)×(N +1)

ΛΛH +

N0
Φη
Es

(10)
(11)

)−1
,

(12)

and Es represents the symbol power. Furthermore, Φη =diag (Φη [0], · · · , Φη [N ]) is a diagonal matrix whose nth element is
)
(
N
N
1 ∑∑
2π(l − m)nΦη [n] =g((l − m)T )exp j
N +1
N +1
m=0
l=0

(0 ≤ n ≤ N ). (13)Finally, by substituting ẑk and ẑk−1 in (11) into yk and yk−1
in (4), the kth symbol s̃k is estimated with the aid of ML
detection.
III. A NALYTICAL E RROR -R ATE B OUND
In [8], the equivalent signal-to-interference-pulse-noise ratio(SINR) value of the received DFTN signals after MMSE-based

POSTPRINT (ACCEPTED VERSION), 10.1109/LSP.2018.2882696

3

FDE is derived. In [14], the analytical BER expression of thedifferentially encoded 16-star-QAM scheme is given. Assistedby these results, we herein derive the analytical error-ratebound of our 16-QAM-aided DFTN signaling scheme.
The estimated 16-star-QAM-aided DFTN symbols ŝ aregiven by [8]ŝ = hQT WΛQ∗ s + QT WQ∗ η= hΓs s + Γn η,(14)where R = E[ηη H ], the ith row and jth column element of
which is given by N0 g((l − m)T ). Hence, the average SINR
value of DFTN signaling after MMSE-FDE is given by
Pd.
PI + Pη(18)

Differential 16-star-QAM is divided into two differential
modulation schemes: two-level differential amplitude shift
keying (2-DASK) and 8-DPSK. In [14], in order to detect coherent Nyquist-criterion differentially encoded 16-star-QAM
symbols, a thresholding scheme, rather than ML detection,
was employed for detecting an amplitude ak . Two thresholding
values are set: ξL = 2/(1 + γ) and ξH = (1 + γ)/2. If
ξL ≤ |yk |/|yk−1 | ≤ ξH , then the amplitude is estimated asâk = âk−1 , that is b4k = 0, and otherwise the amplitude isestimated as âk ̸= âk−1 , that is b4k = 1.
The analytical BER of differential 16-star-QAM is expressed as
1
3
P2-DASK + P8-DPSK ,(19)44
where P2-DASK and P8-DPSK are the analytical BERs of 2-DASK
and 8-DPSK, respectively. Furthermore, letting the amplitudeof the nth symbol be cn , P2-DASK and P8-DPSK are formulatedas
1 1∑∑ϵ(cn , cn−1 )P2-DASK = +2 4 c cP16starQAM =n{1(cn ̸= cn−1 )(22)−1 (cn = cn−1 ).
{}1
ξ 2 − ρ22 /ρ21
P (ξ | cn , cn−1 ) =1+ √,2(ξ 2 + σ22 /σ12 ) − (2ξσ2 /σ1 )2ϵ(cn , cn−1 ) =

[

1
ψ
2
ρ sin ψ1+ + √2π
π 1 − ρ2 cos2 ψ√]
1 + ρ cos ψ
−1
.
× tan
1 − ρ cos ψ(23)F8-DPSK (ψ) =where we have Γn = QT WQ∗ and Γs = Γn G．More
specifically, the desired signals, the FTN-specific ISI, and the
AWGNs are represented, respectively, by sd = hΓd s, sI =h(Γs − Γd )s, and ην = Γn η, where Γd is a diagonal matrix
having diagonal elements of Γs . By considering E[|h|2 ] = 1
and E[ssH ] = IN +1 , the average power of each component isformulated as
{
}
{
}
H
Pd = tr E[sd sH
]
=
tr
Γ
Γ
,
(15)
d d
d
{
}
PI = tr E[sI sH
I ]
{
}
2
H
= tr |Γs | + |Γd |2 − Γs ΓH,(16)d − Γd Γs{}{}Pη = tr E[ην ηνH ] = tr Γn RΓH
,
(17)
n

SINR =

wheren−1[P (ξ = ξH |cn , cn−1 ) − P (ξ = ξL |cn , cn−1 )], (20)
[
(
)]
(
2
π)
3π
P8-DPSK =F8-DPSK ψ = −− F8-DPSK ψ = −
,
3
8
8
(21)

(24)

Furthermore, we have σ22 /σ12 = (c2n + 1/Γ), ρ =cn cn−1 /[(c2n−1 + 1/Γ)(c2n + 1/Γ)]1/2 , and Γ is the average
SINR value. Finally, the analytical BER of our proposed
differential 16-star-QAM-aided DFTN signaling is obtainedby substituting the SINR of (18) into Γ of (19)–(24).
IV. P ERFORMANCE R ESULTS
In this section, we provide our performance results for
16-star-QAM-aided DFTN signaling. The basic parameters
employed in our simulations are as follows. The block lengthof DFTN symbols was set to N = 1024, and a frequency-flatRayleigh fading channel was assumed. The roll-off factor of
the RRC filter was maintained as β =0.3, and the CP length
was assumed to be 2ν = 20. Note that the constellation has
to be optimized, depending on the roll-off factor β employed.
Furthermore, ML detection was employed to calculate the simulated BER curves of the proposed DFTN signaling scheme.
Based on the derived analytical bound of the proposed
DFTN signaling, we optimized the ring ratio γ for each symbol
packing ratio α. We assumed the use of the thresholding
parameters ξL = 2/(1 + γ) and ξH = (1 + γ)/2, accordingto [11]. Fig. 1(a) shows the relationship between ring ratio γ
and the BERs for an Eb /N0 of 20 dB. Symbol packing ratio
α was set to 0.7, 0.8, 0.9, or 1.0. Observe in Fig. 1(a) that the
optimum ring ratio γ changed, depending on symbol packing
ratio α. Furthermore, in Fig. 1(b), the optimum ring ratios γ
for symbol packing ratios of α = 0.7, 0.8, 0.9, and 1.0 wereplotted. As seen in Fig. 1(b), for the low Eb /N0 s, the optimum
ring ratios change depending on symbol packing ratio α. For
the high Eb /N0 s, such as Eb /N0 > 40 dB, the optimum ringratio converges to γ = 2.09 regardless of the α value. Notethat in the range of α < 0.8, the accuracy of the analytical
BER decreases, upon decreasing the α value, similar to [8].
In Fig. 2, we show the numerical results of the achievable
BER performance in the AWGN channel comparing the proposed 16-star-QAM-aided DFTN scheme, the 16-star-QAMaided FTN scheme, the conventional 16-PSK-aided DFTN
scheme, and the conventional 16-PSK-aided FTN scheme.
Moreover, the ring ratio of 16-star-QAM was fixed to γ = 2.1,
and the symbol packing ratio was set as α = 0.9 or 0.8.
Observe in Fig. 2 that the proposed 16-star-QAM-aided DFTN
signaling scheme exhibited a 3-dB penalty over its coherent
counterpart, as expected. This is due to the noise-doubling
effects, which specifically depend on ring ratio γ. Also, thePOSTPRINT (ACCEPTED VERSION), 10.1109/LSP.2018.2882696(a) BER at Eb /N0 = 20 dB4(b) Optimum ring ratio

Fig. 1. Achievable BER performance and optimum ring ratio γ of the proposed 16-point star-QAM-aided DFTN signaling. A time-invariant frequencyflat Rayleigh fading channel was considered; α is the symbol packing ratio.
(a) Achievable BER at the Eb /N0 of 20 dB for each α value; (b) ring ratio
optimized at each Eb /N0 .

Fig. 3. Achievable BERs for the coherent FTN and DFTN signaling schemes
modulated based on 16-star-QAM and 16 PSK. A time-varying frequency-flatRayleigh fading channel was considered at an Eb /N0 of 40 dB. The symbol
packing ratio was set as α = 0.9, and the normalized Doppler frequency(given in parentheses in the legend key) was set as Fd Ts = 1.0 × 10−6 ,1.0 × 10−5 , or 1.0 × 10−4 .

block.1
Finally, as shown in Fig. 3, the effects of a block length on
the achievable BER performance in a time-varying channel
were investigated while considering an Eb /N0 of 40 dB. Also,
the block length N was varied from 26 to 212 , while thenumber of CP was maintained as 2ν = 20. The symbol
packing ratio was set as α = 0.9 and the maximum Doppler
frequency was given by 1 × 10−4 , 1 × 10−5 , and 1 × 10−6 .
Observe in Fig. 3 that the BERs of the proposed 16-star-QAMaided DFTN signaling scheme remained unchanged regardlessof the block length, while those of the coherent counterpart
deteriorated upon increasing the block length.23
V. C ONCLUSIONS
Fig. 2. Achievable BER performance of the coherent FTN and DFTN
schemes, which were modulated based on 16-star-QAM and 16-PSK. An
AWGN channel was considered; α is the symbol packing ratio.

proposed 16-star-QAM-aided DFTN signaling scheme outperformed the conventional 16-PSK-aided DFTN one in terms of
BER. Note that the performance gain owing to the constellation optimization is on the order of a fraction of dB. However,this is not marginal from the physical-layer perspective, since
this optimization can be carried out offline [15].
The above performance results assumed time-invariant
channels; next we investigate the effects of using time-varying
channels. In the time-varying scenario,
the received signals∑of (7) are modified to zk =hsg(k− nT ) + η(kT ),nnnwhere hn (n = 0, · · · , N + 2ν + 1) is the channel coefficient at the nth sample, which is generated according toE[hn h∗n+τ ] = J0 (2πFd T τ ), in which Fd T is the normalized
Doppler frequency and J0 is the zero-order Bessel function of
the first kind. We assumed that, for coherent FTN schemes,
the channel coefficient was updated at the beginning of eachIn this letter, we proposed a novel 16-point star-QAMaided DFTN signaling scheme in order to achieve a better
bandwidth efficiency without the pilot overhead associatedwith channel estimation. The analytical error-rate bound wasderived based on the equivalent SINR value for an MMSEFDE-aided receiver. Furthermore, using this derived bound, a
16-point star-QAM constellation was optimized for our DFTN
signaling scheme in terms of the error-rate performance. Our
performance results revealed that the proposed DFTN scheme
achieved enhanced performance relative to the conventional
16-PSK-modulated DFTN scheme, as well as outperforming
its coherent counterparts, especially in a rapidly time-varying
channel.
1 To expound a little further, for the sake of operating with a low α value,such as α < 0.8, in the DFTN and FTN schemes, the use of powerfulchannel-coding scheme is needed, as mentioned in [6, 13].
2 Note that the proposed DFTN scheme is not directly applicable tothe frequency-selective channel, similar to the conventional differentiallymodulated systems.
3 Note that while our DFTN scheme was optimized for the quasi-static channel, that employing the same parameters exhibited a good BER performancein the time-varying channel.

POSTPRINT (ACCEPTED VERSION), 10.1109/LSP.2018.2882696

R EFERENCES[1] J. Salz, “Optimum mean-square decision feedback equalization,” Bell
System Technical Journal, vol. 52, no. 8, pp. 1341–1373, 1973.
[2] J. Mazo, “Faster-than-Nyquist signaling,” Bell System Technical Journal,
vol. 54, no. 8, pp. 1451–1462, 1975.
[3] S. Sugiura, “Frequency-domain equalization of faster-than-Nyquist signaling,” IEEE Wireless Communications Letters, vol. 2, no. 5, pp. 555–
558, Oct. 2013.
[4] R. Dinis, B. Cunha, F. Ganhao, L. Bernardo, R. Oliveira, and P. Pinto,“A hybrid ARQ scheme for faster than Nyquist signaling with iterative frequency-domain detection,” in IEEE 81st Vehicular Technology
Conference (VTC2015-Spring), May 2015, pp. 1–5.
[5] W. Yuan, N. Wu, H. Wang, and J. Kuang, “Variational inferencebased frequency-domain equalization for faster-than-Nyquist signaling
in doubly selective channels,” IEEE Signal Processing Letters, vol. 23,
no. 9, pp. 1270–1274, Sept. 2016.
[6] T. Ishihara and S. Sugiura, “Iterative frequency-domain joint channel
estimation and data detection of faster-than-Nyquist signaling,” IEEE
Transactions on Wireless Communications, vol. 16, no. 9, pp. 6221–
6231, Sept. 2017.
[7] Q. Shi, N. Wu, X. Ma, and H. Wang, “Frequency-domain joint channel estimation and decoding for faster-than-Nyquist signaling,” IEEE
Transactions on Communications, 2017, in press.
[8] T. Ishihara and S. Sugiura, “Differential faster-than-Nyquist signaling,”
IEEE Access, vol. 6, pp. 4199–4206, 2018.

5[9] C.-D. Chung, “Differentially amplitude and phase-encoded QAM forthe correlated rayleigh-fading channel with diversity reception,” IEEE
Transactions on Communications, vol. 45, no. 3, pp. 309–321, Mar.
1997.
[10] D. Liang, S. X. Ng, and L. Hanzo, “Soft-decision star-QAM aided
BICM-ID,” IEEE Signal Processing Letters, vol. 18, no. 3, pp. 169–
172, Mar. 2011.
[11] W. T. Webb, L. Hanzo, and R. Steele, “Bandwidth efficient QAM
schemes for Rayleigh fading channels,” IEE Proceedings I - Communications, Speech and Vision, vol. 138, no. 3, pp. 169–175, June 1991.
[12] T. Suzuki and T. Mizuno, “Multiple-symbol differential detection
scheme for differential amplitude modulation,” in Mobile Communications Advanced Systems and Components, C. G. Günther, Ed. Berlin,
Heidelberg: Springer Berlin Heidelberg, 1994, pp. 196–207.
[13] S. Sugiura and L. Hanzo, “Frequency-domain-equalization-aided iterative detection of faster-than-Nyquist signaling,” IEEE Transactions on
Vehicular Technology, vol. 64, no. 5, pp. 2122–2128, 2015.
[14] F. Adachi and M. Sawahashi, “Performance analysis of various 16 level
modulation schemes under Rayleigh fading,” Electronics Letters, vol. 28,
no. 17, pp. 1579–1581, 1992.
[15] S. Sugiura and L. Hanzo, “On the joint optimization of dispersion
matrices and constellations for near-capacity irregular precoded spacetime shift keying,” IEEE Transactions on Wireless Communications,vol. 12, no. 1, pp. 380–387, Jan. 2013.