# An Improved Data-Driven Decision Feedback Receiver via Deep Unfolding

Authors: Song Liu; Linchang Zhao; Tao Song
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

The proposed decision feedback receiver (DFR) is an end-to-end data-driven iterative receiver, and the performance gain is achieved from iterations. However, the mismatch problem between the training set and test set exists in the DFR training, and thus performance degradation, slow convergence speed, and oscillation are introduced. On the other hand, deep unfolding using parameter sharing is a practical method to reduce the model parameter number and improve the training efficiency, but the problem whether the parameter sharing will cause performance degradation is rarely considered. In this work, we generally discuss and analyze these two problems, and solution to solve the problem or conditions that the problem no longer exists is then introduced. We give the improvements to address the mismatch problem in the DFR, and thus we propose an improved-DFR via deep unfolding. The improved-DFRs with and without parameter sharing, namely, DFR-I and DFR-IS, are both developed with low computation complexity and model complexity and can be executed by parallel processing. Besides, the practical training tricks and performance analysis including computation complexity and model complexity are given. In the experiments, the improved-DFRs outperform the DFR in various scenarios, in terms of convergence speed and symbol error rate. The simulation results also show that the DFR-IS is easier to train, and the slight performance loss can be reduced by increasing model complexity, in comparison to DFR-I.


---

## Body

## 1. Introduction

Machine learning (ML) or deep learning (DL) [1–5] is promising for wireless communications. The communication system is typically model-driven and thus the corresponding design is derived by domain knowledge. Meanwhile, the ML-enabled approach is data driven, and it learns to optimize the system design through data training. ML is applicable for end-to-end optimization problems in complex scenarios with inaccurate and/or intractable models [6, 7]. Neural network (NN) is a fundamental model of ML. As a universal function approximator, NN has been widely studied in a range of optimization problems in wireless communications, such as signal detection [8–11], degree of arrival (DOA) estimation [12, 13], beam prediction [14], and resource allocation [15, 16]. Besides, the rapid development of massively parallel processing hardwares with distributed memory guarantees the deployment and execution of NN-based algorithms to be fast and efficient.Deep unfolding [17] is proposed as a combination of the data-driven NN-based learning approaches and the model-based iterative algorithms, and it has been widely investigated in the wireless communication community [6, 18, 19]. In deep unfolding, the iterations are unfolded into a layer-wise structure analogous to a NN, and the model parameters across layers are untied to obtain NN-like architectures which are learnable. The resulting formula makes an iterative algorithm to be capable of data learning and thus a better performance can be achieved [20]. In some cases where the signal dimension is high and the mathematical model is unpresented, the iterative algorithms are developed in a data-driven manner, and thus the scale of trainable parameters in an unfolded NN is extremely large [21, 22] and the training is challenging. Parameter sharing is a practical technology which allows the parameters across different layers to be the same and thus reduces the number of trainable parameters [9, 22, 23]. On the other hand, without parameter sharing, the number of samples with respect to the corresponding trainable parameter is in inverse proportion to the number of layers. Therefore, the decrease in model complexity and the increase in sample complexity are beneficial to overcome the overfitting problem and also improve the learning efficiency. Intuitively, the reduction of parameters in the whole unfolded NN will lead to a decrease in model approximation quality. But strictly speaking, whether the parameter sharing would lead to performance degradation is rarely investigated.Considering signal detection, Ye et al. presented initial results in deep learning for signal detection in orthogonal frequency-division multiplexing (OFDM) systems [8]. In this letter, they exploit deep learning to handle wireless OFDM channels in an end-to-end manner. Different from existing OFDM receivers that first estimate channel state information (CSI) explicitly and then detect/recover the transmitted symbols using the estimated CSI, the proposed deep learning-based approach estimates CSI implicitly and recovers the transmitted symbols directly. Samuel et al. considered the use of deep neural networks in the context of Multiple-Input-Multiple-Output (MIMO) detection; a model and data-driven neural network architecture suitable for this detection task is proposed [24]. Furthermore, a MIMO detector which is specially designed by unfolding an iterative algorithm and adding some trainable parameters is proposed in [9]. Since the number of trainable parameters is much fewer than the data-driven DL-based signal detector, the model-driven DL-based MIMO detector can be rapidly trained with a much smaller data set.In [25], a point-to-point band-pass wireless communication system is considered. The transmitted symbols are mapped by m-ary phase position shift keying (MPPSK) modulation [26], and the infinite impulse response (IIR) band-pass filters [27] are deployed at both the transmitter and receiver, to shape the transmitting signal and obviate the disturbances, respectively. However, the filters also introduce inter-symbol-interference (ISI), waveform distortions, and nonwhite noise. To address these issues, a matched filter, an equalizer, and a demodulator are usually required at the receiver. The ISI becomes severe when the allocated bandwidth is narrowed, and equalization becomes the key issue in this receiver. However, equalizer design in such a system is challenging. First, the system is IIR, but the typical equalizers are developed with the assumption that the systems are finite impulse response (FIR) [28]. Second, the symbols are mathematically described as scalars in a typical equalizer; meanwhile, the symbols modulated by MPPSK are formulated as vectors in the time domain. Third, the block-by-block system design is fussy. Due to these issues, the receiver design is intractable and thus we proposed an end-to-end NN-based receiver which iteratively estimates the transmitted symbols, namely the decision feedback receiver (DFR). Simulation results showed that, after several iterations the DFR had better detection performance than the receiver without feedbacks. However, we also found that the first solution of DFR performed worse than the receiver without feedbacks, but theoretically they are the same. Besides, the detection instability along iteration times occurred when the soft information of both posterior and previous adjacent symbols was utilized as feedbacks. These existing problems are needed to be solved.In this work, we study the principle behind the performance degradation of DFR in a general way. Based on the theoretical analysis, we propose an improved-DFR to address the existing problems in DFR. Specially speaking, this work makes the following contributions:(i)
We consider a mismatch between the training set and test set, and point out that the increased test error is caused bydata missing and data divergence. We also give the upper and lower bounds of the test error. Then, we respectively introduce the solution to eliminate the error caused by data missing and the conditions that inherent data divergence no longer exists.(ii)
We consider training one single model on an integration of multiple training sets and point out that the increased test error is caused bydata divergence. We also introduce the conditions that data divergence no longer exists. Based on these conclusions, under sufficient model complexity assumption, we prove that data divergence does not exist in deep unfolding using parameter sharing. Furthermore, we use the Markov decision process (MDP) [29] to describe the iterative algorithm with insufficient model complexity, and the condition of data divergence nonexistence is given.(iii)
We point out the existing problems in DFR are introduced by the mismatch between the data sets. To address the mismatch problem, we further propose the improved-DFR via deep unfolding. The improvements include: the DFR is unfolded to several sub-NNs, and the intermediate solutions can be obtained during training; each sub-NN is trained with its parameter, and the possibly existingdata divergence is avoided; the soft information of both posterior and previous adjacent symbols are utilized as feedbacks, and the compromised method to guarantee stability becomes unnecessary.(iv)
The improved-DFR is an end-to-end data-driven receiver, and it has low computation complexity and model complexity and can be executed by parallel processing. We also propose a clip technique to solve the numeric overflow problem in practical training. In the experiments, the improved-DFRs outperform the DFR in various scenarios, in terms of convergence speed and symbol error rate (SER). The simulation results also show that the DFR-IS is easier to train, and the slight performance loss can be reduced by increasing model complexity, in comparison to DFR-I.The remainder of this study is organized as follows. The theoretical analysis including the test error brought by the mismatch problem, the integration of multiple training sets, and the parameter sharing are given in Section 2. The DFR and the proposed improved-DFR are presented in Section3. The simulation results of the improved-DFR with and without parameter sharing and DFR are demonstrated in Section 4. Conclusions are presented in Section 5.


## 2. Theoretical Analysis

### 2.1. Preliminary

We begin by establishing the models of the data set. We assume that the potential mapping in the data set is a surjection in which every element of the image space is a value for some members of the domain, and the mapping also satisfies Lipschitz continuity. The mappingf in the training set is expressed as(1)f:x∼X↦y∼Y,where the input vector x following the random distribution X is an input random variable vector, and the output vector y following the random distribution Y. The probability density function (PDF) of X is px. We sample the input–output pairs of f independently and identically, then we get the training set Strain=xn,ynn=1Ntrain, where the subscript “train” denotes the training set, and Ntrain is the number of training samples. Similarly, the mapping g in the test set is defined as(2)g:x∼X′↦y∼Y′,where X′ is a random variable with PDF being qx. We sample the input–output pairs of g and get the testing set Stest=xn,ynn=1Ntest, where the subscript “test” denotes the testing set, and Ntest is the number of testing samples.Second, we define a spaceDp∩q, an intersection of support sets of functions px and qx:(3)Dp∩q=x|px>0,qx>0.We also define a spaceDp/q which is a complement of the support set of qx in the support set of px, and similarly we have a space Dq/p. The two spaces are written as follows:(4)Dp/q=x|px>0,qx=0,Dq/p=x|px=0,qx>0,Figure1 shows an illustration of the spaces px and qx with a one-dimensional variable x. Particularly, Dp∩q is an intersection of support sets of functions p and q, Dp/q is a complement of support set of q in the support set of p, and Dq/p is a complement of support set of p in the support set of q.Figure 1
The illustration of the spacesDp∩q, Dp/q and Dq/p with one-dimensional variable x.


### 2.2. Mismatch between Training Set and Test Set

In ML, the potential mappings in the training set and the test set are usually the same, i.e.,f=g, and they follow the same distribution, i.e., px=qx. Using a training set where Ntrain⟶∞, a parameterized model fθ of sufficient complexity can be sufficiently trained, then we have fθ⟶f=g. This indicates that the learned model fθ has zero error on the test set. First, we investigate the mismatch problem between the training set and the test set: when there exists a difference between the distributions of the data sets, i.e., KLp‖q>0 where KL⋅ is the Kullback–Leibler divergence [2], and thus f≠g, then how will the learned model perform on the test set.Without loss of generality, we use some distance functiond⋅ as an error function, and the expected error on the test set can be expressed as(5)Lf,g=∫qxdfx,gxdx,=∫Dp∩q∪Dq/pqxdfx,gxdx=∫Dp∩qqxdfx,gxdx+∫Dq/pqxdfx,xdx≤∫Dp∩qqxdmaxDp∩qf,gdx+∫Dq/pqxdfx,gxdx≤∫Dp∩qqxdmaxDp∩qf,gdx+∫Dq/pqxdmaxDq/pf,gdx=dmaxDp∩qf,g∫Dp∩qqxdx+dmaxDq/pf,g∫Dq/pqxdx=dmaxDp∩qf,gDqp∩q+dmaxDq/pf,gDq/p,where Dqp∩q=∫Dp∩qqxdx., Dq/p=∫Dq/pqxdx, and(6)dmaxDp∩qf,g=maxx∈Dp∩qdfx,gx,dmaxDq/pf,g=maxx∈Dq/pdfx,gx.The upper bound of the test error is given in (5), and it indicates that the error is composed of two parts. First in the space Dp∩q, the images of the same variable are different, i.e., gx≠fx, we call this inherent error as data divergence, and it cannot be cancelled. The other part of the error comes from the input space Dq/p, which only shows in the test set and is unpresented in the training set. This error is caused by data missing, and the model cannot capture the proper mapping during learning. Particularly, the data missing is different from the generalization error. By increasing sample complexity, data divergence cannot be reduced but generalization error can be alleviated. However, the samples which follow the distribution of the test set can be added to the training set, and then the data missing can be removed. We consider three cases. First, after proper training with added samples, the upper bound of the test error can be reduced to(7)supLf,g=dmaxDp∩qf,gDqp∩q.Second, when any of the following two conditions are satisfied:(8)Dp∩q=∅,(9)gx=fx,∀x∈Dp∩q,meanwhile, if the added samples are unavailable, then the upper bound of the test error is(10)supLf,g=dmaxDq/pf,gDq/p,Third, when the learned model eliminates thedata missing with added samples, and the data set satisfied any of conditions (8) and (9), then supLf,g=0, According to the nonnegativity of PDF and distance function, the upper bound is also the lower bound.


### 2.3. Integration of Multiple Training Sets

We have quantitatively analyzed the test error when the training set and test set mismatch, and in this subsection we focus on the situation where multiple training sets which follow different distributions, integrate into a new training set. First, we consider two training sets, i.e.,K=2, and their potential mappings are f and g, respectively. We assume that f is sampled with probability u∈0,1 in the integrated training set Strain, and the test set Stest follows the same distribution as the training set. Therefore, the expected errors on the training set Strain and the test set Stest are the same. The model h is trained with Strain, and the corresponding error can be written as(11)Lh,f,g=uLh,f+1−uLh,g,=u∫Dp∩qpxdhx,fxdx+u∫Dp/qpxdhx,fxdx,+1−u∫Dp∩qqxdhx,gxdx+1−u∫Dq/pqxdhx,gxdx,=u∫Dp∩qpxdhx,fxdx+1−u∫Dp∩qqxdhx,gxdx,=∫Dp∩qu⋅pxdhx,fx+1−uqxdhx,gxdx.According to (11), ∀x∈Dp∩q, ∃h∗x:(12)h∗x=argminhu⋅pxdhx,fx+1−uqxdhx,gx.Especially, according to the symmetry and trigonometric inequality ofd⋅, when u=qx/px+qx where u is a function of x, then we have(13)h∗x=fx+gx2.Plug (13) into (11), and (11) can be rewritten as(14)Lh∗,f,g=∫Dp∩q2pxqxpx+qxdfx+gx2,fx.More generally, after sufficient training, the learned functionh∗ can be achieved as(15)h∗x=fx,x∈Dp/q,h∗x,x∈Dp∩q,gx,x∈Dp/q.Plug (15) into (11), and we have(16)Lh∗,f,g=∫Dp∩qu⋅pxdh∗x,fx+1−uqxdh∗x,gxdx,≤maxdmaxDp∩qh∗,f,dmaxDp∩qh∗,g∫Dp∩qu⋅px+1−uqxdx,=maxdmaxDp∩qh∗,f,dmaxDp∩qh∗,g×∫Dp∩q2pxqxpx+qxdx,≤maxdmaxDp∩qh∗,f,dmaxDp∩qh∗,g×∫Dp∩qpx+qxdx=maxdmaxDp∩qh∗,f,dmaxDp∩qh∗,g×Dpp∩q+Dqp∩q.where(17)dmaxDp∩qh∗,f=maxx∈Dp∩qdh∗x,fx,dmaxDp∩qh∗,g=maxx∈Dp∩qdh∗x,gx.The upper bound of error is given in (16). Data missing no longer exists in this situation, but data divergence arises from the integration. Similarly, when the data set satisfies condition (8) or (9), then the upper bound and the lower bound of error are zeros, i.e., supLh,f,g=infLh,f,g=0.The above conclusions can be extended toK>2. Given a set of mapping functions Sf=fkk=1K, and the element mapping is expressed as(18)fk:x∼Xk↦y∼Yk,where we use pkx to denote the PDF of Xk, ∀i,j∈1,…,K,i≠j, when any of the following two conditions:(19)Dpi∩pj=∅,(20)fix=fjx,∀x∈Dpi∩pj,are satisfied, and we train model h of sufficient complexity with the sufficient large training set, then the expected error on the data set reaches the lower bound, i.e., Lh,Sf=0.In summary, thedata missing can be removed by the solution that adding the missing samples to the training set. Moreover, the condition (19) or (20) guarantees that the mapping in the integrated training set is still a surjection, and then the inherent data divergence no longer exists.


### 2.4. Deep Unfolding

In deep unfolding, an iteration algorithm can be unfolded as follows:(21)fθk:x,ϕk−1↦ϕk,where k is the iteration index, x is the input, ϕk is the current iterative solution and also part of the input in the next iteration, and θk is the parameter of layer k. In the first iteration, ϕ0 is some initial value. The target solution of the iteration algorithm is y. Given an algorithm with Kmax iteration times, the parameter set of the corresponding model is θkk=1Kmax. In some cases, the parameters in different layers are shared, i.e., θk=θ,∀k.We discuss the training procedures with and without parameter sharing here. First, given a parameter setθkk=1Kmax and a set of training sets Strainkk=1Kmax where the element training set Straink is provided for parameter θk, and the element sample in Straink is x,ϕk−1,y. Without parameter sharing, there are Kmax layers, and each layer is trained with its training set with respect to its parameter. Second, we consider the training case with parameter sharing. According to the discussions in subsection 2.2 and subsection 2.3, a single model is used to approximate multiple potential mappings, and thus the corresponding training set must include samples of all these mappings to eliminate data missing. Hence, an integrated training set Strain=Strain1∪⋯∪StrainKmax is provided for the model with a shared parameter θ. Using parameter sharing, only one model is trained with respect to the shared parameter on an integrated training set.Several benefits can be gained from parameter sharing. Assuming that the amounts of the parameter in different layers are the same, then the total amount of model parameter is reduced to1/Kmax times, but the amount of samples with respect to the corresponding parameter is increased to Kmax times. The model complexity and parameter redundancy are reduced, but the sample complexity is increased, and thus the overfitting can be alleviated and the learning efficiency can be improved.On the other hand, some negative issues can arise from parameter sharing. For example, the decrease in parameter number also leads to a decrease in model approximation quality. When we consider parameter sharing, the model complexity can be appropriately increased, because that the union spaceDp1∪⋯∪DpKmax is larger than any subspace Dpk, where pk is the PDF of input x,ϕk−1. Moreover, there is always a trade-off between the model approximation quality and computation efficiency. Normally, a proper NN scale can be determined through simulations. Most importantly, we consider such a problem: does the test error caused by data divergence exist in deep unfolding with parameter sharing?


#### 2.4.1. Sufficient Model Complexity

According to the introduced formulation and training of deep unfolding, given any inputx,ϕk−1 and output ϕk in any layer k, then the target solutions are the same, i.e., Yk=y,∀k. This indicates that when the model complexity and sample complexity are both sufficient, after sufficient training, then we have(22)fθk:x,ϕk−1↦ϕk⟶y,∀x,ϕk−1,∀k∈1,…,Kmax.Equation (22) shows that deep unfolding satisfies condition (20), and the potential mapping in the integrated training set is still a surjection. Therefore, the data divergence does not exist in deep unfolding with parameter sharing, and the corresponding upper bound of the test error is zero.In fact, the assumption of sufficient model complexity is impractical. Due to the extreme complexity of potential functionf, the practical parameterized model fθ usually cannot approximate f with arbitrary precision. Therefore, we resort to iterative methods which approach the optimal solution step-by-step, instead of reaching the optimum by a single step. Therefore, the data divergence still possibly exists due to the insufficient model complexity.


#### 2.4.2. Insufficient Model Complexity

We resort to the MDP to describe the iterative optimization in deep unfolding with insufficient model complexity. MDP is a typical model in reinforcement learning [29], which concerns about an agent interacting with an environment in the single-agent case. In each interaction, the agent takes action a by policy π using the observed state s, then receives a feedback reward r and an updated state s′ from the environment. The agent aims to find an optimal policy to maximize the cumulative reward over the continuous interactions.The inference process of iterative algorithms can be described as MDP, as shown in Figure2. The agent policy is the model function π=f. At the time k, the state is sk=x,ϕk−1, the action is produced by the policy (namely the model function), and it is ak=ϕk. The reward is some negative distance function −dϕk,ϕ∗ where ϕ∗ denotes the optimum solution with respect to x. The environment is then transferred to the updated state sk+1=x,ϕk. In such formulation, the iterative inference is in consistent with the MDP.Figure 2
An illustrative interaction of an MDP.We can prove that: when the performance of the intermediate solution−dϕk,ϕ∗ is monotonously increasing, then the model f is a surjection and thus the data divergence does not exist.Proof.
Given any inputx and intermediate solution ϕi, ϕj,i,j∈0,1,…,Kmax,i≠j. When −dϕi,ϕ∗<−dϕj,ϕ∗, then according to the monotonicity of the distance function, we have ϕi≠ϕj.
Given a solution setϕkk=0Kmax where the elements satisfy −dϕ0,ϕ∗<⋯<−dϕKmax,ϕ∗, and thus we have ϕi≠ϕj,i,j∈0,1,…,Kmax,i≠j. The mapping of any input x,ϕk−1 is a unique ϕk. Therefore, the mapping f is a bijection and also a surjection, and the data divergence does not exist.The above conclusion is derived without the assumption of sufficient model complexity. When the monotonously increasing condition is not satisfied, the oscillation possibly occurs. Besides, when the monotonously increasing condition is satisfied, thedata divergence does not exist in the situation where the model parameter is not shared and the model complexity is limited.


## 2.1. Preliminary

We begin by establishing the models of the data set. We assume that the potential mapping in the data set is a surjection in which every element of the image space is a value for some members of the domain, and the mapping also satisfies Lipschitz continuity. The mappingf in the training set is expressed as(1)f:x∼X↦y∼Y,where the input vector x following the random distribution X is an input random variable vector, and the output vector y following the random distribution Y. The probability density function (PDF) of X is px. We sample the input–output pairs of f independently and identically, then we get the training set Strain=xn,ynn=1Ntrain, where the subscript “train” denotes the training set, and Ntrain is the number of training samples. Similarly, the mapping g in the test set is defined as(2)g:x∼X′↦y∼Y′,where X′ is a random variable with PDF being qx. We sample the input–output pairs of g and get the testing set Stest=xn,ynn=1Ntest, where the subscript “test” denotes the testing set, and Ntest is the number of testing samples.Second, we define a spaceDp∩q, an intersection of support sets of functions px and qx:(3)Dp∩q=x|px>0,qx>0.We also define a spaceDp/q which is a complement of the support set of qx in the support set of px, and similarly we have a space Dq/p. The two spaces are written as follows:(4)Dp/q=x|px>0,qx=0,Dq/p=x|px=0,qx>0,Figure1 shows an illustration of the spaces px and qx with a one-dimensional variable x. Particularly, Dp∩q is an intersection of support sets of functions p and q, Dp/q is a complement of support set of q in the support set of p, and Dq/p is a complement of support set of p in the support set of q.Figure 1
The illustration of the spacesDp∩q, Dp/q and Dq/p with one-dimensional variable x.


## 2.2. Mismatch between Training Set and Test Set

In ML, the potential mappings in the training set and the test set are usually the same, i.e.,f=g, and they follow the same distribution, i.e., px=qx. Using a training set where Ntrain⟶∞, a parameterized model fθ of sufficient complexity can be sufficiently trained, then we have fθ⟶f=g. This indicates that the learned model fθ has zero error on the test set. First, we investigate the mismatch problem between the training set and the test set: when there exists a difference between the distributions of the data sets, i.e., KLp‖q>0 where KL⋅ is the Kullback–Leibler divergence [2], and thus f≠g, then how will the learned model perform on the test set.Without loss of generality, we use some distance functiond⋅ as an error function, and the expected error on the test set can be expressed as(5)Lf,g=∫qxdfx,gxdx,=∫Dp∩q∪Dq/pqxdfx,gxdx=∫Dp∩qqxdfx,gxdx+∫Dq/pqxdfx,xdx≤∫Dp∩qqxdmaxDp∩qf,gdx+∫Dq/pqxdfx,gxdx≤∫Dp∩qqxdmaxDp∩qf,gdx+∫Dq/pqxdmaxDq/pf,gdx=dmaxDp∩qf,g∫Dp∩qqxdx+dmaxDq/pf,g∫Dq/pqxdx=dmaxDp∩qf,gDqp∩q+dmaxDq/pf,gDq/p,where Dqp∩q=∫Dp∩qqxdx., Dq/p=∫Dq/pqxdx, and(6)dmaxDp∩qf,g=maxx∈Dp∩qdfx,gx,dmaxDq/pf,g=maxx∈Dq/pdfx,gx.The upper bound of the test error is given in (5), and it indicates that the error is composed of two parts. First in the space Dp∩q, the images of the same variable are different, i.e., gx≠fx, we call this inherent error as data divergence, and it cannot be cancelled. The other part of the error comes from the input space Dq/p, which only shows in the test set and is unpresented in the training set. This error is caused by data missing, and the model cannot capture the proper mapping during learning. Particularly, the data missing is different from the generalization error. By increasing sample complexity, data divergence cannot be reduced but generalization error can be alleviated. However, the samples which follow the distribution of the test set can be added to the training set, and then the data missing can be removed. We consider three cases. First, after proper training with added samples, the upper bound of the test error can be reduced to(7)supLf,g=dmaxDp∩qf,gDqp∩q.Second, when any of the following two conditions are satisfied:(8)Dp∩q=∅,(9)gx=fx,∀x∈Dp∩q,meanwhile, if the added samples are unavailable, then the upper bound of the test error is(10)supLf,g=dmaxDq/pf,gDq/p,Third, when the learned model eliminates thedata missing with added samples, and the data set satisfied any of conditions (8) and (9), then supLf,g=0, According to the nonnegativity of PDF and distance function, the upper bound is also the lower bound.


## 2.3. Integration of Multiple Training Sets

We have quantitatively analyzed the test error when the training set and test set mismatch, and in this subsection we focus on the situation where multiple training sets which follow different distributions, integrate into a new training set. First, we consider two training sets, i.e.,K=2, and their potential mappings are f and g, respectively. We assume that f is sampled with probability u∈0,1 in the integrated training set Strain, and the test set Stest follows the same distribution as the training set. Therefore, the expected errors on the training set Strain and the test set Stest are the same. The model h is trained with Strain, and the corresponding error can be written as(11)Lh,f,g=uLh,f+1−uLh,g,=u∫Dp∩qpxdhx,fxdx+u∫Dp/qpxdhx,fxdx,+1−u∫Dp∩qqxdhx,gxdx+1−u∫Dq/pqxdhx,gxdx,=u∫Dp∩qpxdhx,fxdx+1−u∫Dp∩qqxdhx,gxdx,=∫Dp∩qu⋅pxdhx,fx+1−uqxdhx,gxdx.According to (11), ∀x∈Dp∩q, ∃h∗x:(12)h∗x=argminhu⋅pxdhx,fx+1−uqxdhx,gx.Especially, according to the symmetry and trigonometric inequality ofd⋅, when u=qx/px+qx where u is a function of x, then we have(13)h∗x=fx+gx2.Plug (13) into (11), and (11) can be rewritten as(14)Lh∗,f,g=∫Dp∩q2pxqxpx+qxdfx+gx2,fx.More generally, after sufficient training, the learned functionh∗ can be achieved as(15)h∗x=fx,x∈Dp/q,h∗x,x∈Dp∩q,gx,x∈Dp/q.Plug (15) into (11), and we have(16)Lh∗,f,g=∫Dp∩qu⋅pxdh∗x,fx+1−uqxdh∗x,gxdx,≤maxdmaxDp∩qh∗,f,dmaxDp∩qh∗,g∫Dp∩qu⋅px+1−uqxdx,=maxdmaxDp∩qh∗,f,dmaxDp∩qh∗,g×∫Dp∩q2pxqxpx+qxdx,≤maxdmaxDp∩qh∗,f,dmaxDp∩qh∗,g×∫Dp∩qpx+qxdx=maxdmaxDp∩qh∗,f,dmaxDp∩qh∗,g×Dpp∩q+Dqp∩q.where(17)dmaxDp∩qh∗,f=maxx∈Dp∩qdh∗x,fx,dmaxDp∩qh∗,g=maxx∈Dp∩qdh∗x,gx.The upper bound of error is given in (16). Data missing no longer exists in this situation, but data divergence arises from the integration. Similarly, when the data set satisfies condition (8) or (9), then the upper bound and the lower bound of error are zeros, i.e., supLh,f,g=infLh,f,g=0.The above conclusions can be extended toK>2. Given a set of mapping functions Sf=fkk=1K, and the element mapping is expressed as(18)fk:x∼Xk↦y∼Yk,where we use pkx to denote the PDF of Xk, ∀i,j∈1,…,K,i≠j, when any of the following two conditions:(19)Dpi∩pj=∅,(20)fix=fjx,∀x∈Dpi∩pj,are satisfied, and we train model h of sufficient complexity with the sufficient large training set, then the expected error on the data set reaches the lower bound, i.e., Lh,Sf=0.In summary, thedata missing can be removed by the solution that adding the missing samples to the training set. Moreover, the condition (19) or (20) guarantees that the mapping in the integrated training set is still a surjection, and then the inherent data divergence no longer exists.


## 2.4. Deep Unfolding

In deep unfolding, an iteration algorithm can be unfolded as follows:(21)fθk:x,ϕk−1↦ϕk,where k is the iteration index, x is the input, ϕk is the current iterative solution and also part of the input in the next iteration, and θk is the parameter of layer k. In the first iteration, ϕ0 is some initial value. The target solution of the iteration algorithm is y. Given an algorithm with Kmax iteration times, the parameter set of the corresponding model is θkk=1Kmax. In some cases, the parameters in different layers are shared, i.e., θk=θ,∀k.We discuss the training procedures with and without parameter sharing here. First, given a parameter setθkk=1Kmax and a set of training sets Strainkk=1Kmax where the element training set Straink is provided for parameter θk, and the element sample in Straink is x,ϕk−1,y. Without parameter sharing, there are Kmax layers, and each layer is trained with its training set with respect to its parameter. Second, we consider the training case with parameter sharing. According to the discussions in subsection 2.2 and subsection 2.3, a single model is used to approximate multiple potential mappings, and thus the corresponding training set must include samples of all these mappings to eliminate data missing. Hence, an integrated training set Strain=Strain1∪⋯∪StrainKmax is provided for the model with a shared parameter θ. Using parameter sharing, only one model is trained with respect to the shared parameter on an integrated training set.Several benefits can be gained from parameter sharing. Assuming that the amounts of the parameter in different layers are the same, then the total amount of model parameter is reduced to1/Kmax times, but the amount of samples with respect to the corresponding parameter is increased to Kmax times. The model complexity and parameter redundancy are reduced, but the sample complexity is increased, and thus the overfitting can be alleviated and the learning efficiency can be improved.On the other hand, some negative issues can arise from parameter sharing. For example, the decrease in parameter number also leads to a decrease in model approximation quality. When we consider parameter sharing, the model complexity can be appropriately increased, because that the union spaceDp1∪⋯∪DpKmax is larger than any subspace Dpk, where pk is the PDF of input x,ϕk−1. Moreover, there is always a trade-off between the model approximation quality and computation efficiency. Normally, a proper NN scale can be determined through simulations. Most importantly, we consider such a problem: does the test error caused by data divergence exist in deep unfolding with parameter sharing?


### 2.4.1. Sufficient Model Complexity

According to the introduced formulation and training of deep unfolding, given any inputx,ϕk−1 and output ϕk in any layer k, then the target solutions are the same, i.e., Yk=y,∀k. This indicates that when the model complexity and sample complexity are both sufficient, after sufficient training, then we have(22)fθk:x,ϕk−1↦ϕk⟶y,∀x,ϕk−1,∀k∈1,…,Kmax.Equation (22) shows that deep unfolding satisfies condition (20), and the potential mapping in the integrated training set is still a surjection. Therefore, the data divergence does not exist in deep unfolding with parameter sharing, and the corresponding upper bound of the test error is zero.In fact, the assumption of sufficient model complexity is impractical. Due to the extreme complexity of potential functionf, the practical parameterized model fθ usually cannot approximate f with arbitrary precision. Therefore, we resort to iterative methods which approach the optimal solution step-by-step, instead of reaching the optimum by a single step. Therefore, the data divergence still possibly exists due to the insufficient model complexity.


### 2.4.2. Insufficient Model Complexity

We resort to the MDP to describe the iterative optimization in deep unfolding with insufficient model complexity. MDP is a typical model in reinforcement learning [29], which concerns about an agent interacting with an environment in the single-agent case. In each interaction, the agent takes action a by policy π using the observed state s, then receives a feedback reward r and an updated state s′ from the environment. The agent aims to find an optimal policy to maximize the cumulative reward over the continuous interactions.The inference process of iterative algorithms can be described as MDP, as shown in Figure2. The agent policy is the model function π=f. At the time k, the state is sk=x,ϕk−1, the action is produced by the policy (namely the model function), and it is ak=ϕk. The reward is some negative distance function −dϕk,ϕ∗ where ϕ∗ denotes the optimum solution with respect to x. The environment is then transferred to the updated state sk+1=x,ϕk. In such formulation, the iterative inference is in consistent with the MDP.Figure 2
An illustrative interaction of an MDP.We can prove that: when the performance of the intermediate solution−dϕk,ϕ∗ is monotonously increasing, then the model f is a surjection and thus the data divergence does not exist.Proof.
Given any inputx and intermediate solution ϕi, ϕj,i,j∈0,1,…,Kmax,i≠j. When −dϕi,ϕ∗<−dϕj,ϕ∗, then according to the monotonicity of the distance function, we have ϕi≠ϕj.
Given a solution setϕkk=0Kmax where the elements satisfy −dϕ0,ϕ∗<⋯<−dϕKmax,ϕ∗, and thus we have ϕi≠ϕj,i,j∈0,1,…,Kmax,i≠j. The mapping of any input x,ϕk−1 is a unique ϕk. Therefore, the mapping f is a bijection and also a surjection, and the data divergence does not exist.The above conclusion is derived without the assumption of sufficient model complexity. When the monotonously increasing condition is not satisfied, the oscillation possibly occurs. Besides, when the monotonously increasing condition is satisfied, thedata divergence does not exist in the situation where the model parameter is not shared and the model complexity is limited.


## 2.4.1. Sufficient Model Complexity

According to the introduced formulation and training of deep unfolding, given any inputx,ϕk−1 and output ϕk in any layer k, then the target solutions are the same, i.e., Yk=y,∀k. This indicates that when the model complexity and sample complexity are both sufficient, after sufficient training, then we have(22)fθk:x,ϕk−1↦ϕk⟶y,∀x,ϕk−1,∀k∈1,…,Kmax.Equation (22) shows that deep unfolding satisfies condition (20), and the potential mapping in the integrated training set is still a surjection. Therefore, the data divergence does not exist in deep unfolding with parameter sharing, and the corresponding upper bound of the test error is zero.In fact, the assumption of sufficient model complexity is impractical. Due to the extreme complexity of potential functionf, the practical parameterized model fθ usually cannot approximate f with arbitrary precision. Therefore, we resort to iterative methods which approach the optimal solution step-by-step, instead of reaching the optimum by a single step. Therefore, the data divergence still possibly exists due to the insufficient model complexity.


## 2.4.2. Insufficient Model Complexity

We resort to the MDP to describe the iterative optimization in deep unfolding with insufficient model complexity. MDP is a typical model in reinforcement learning [29], which concerns about an agent interacting with an environment in the single-agent case. In each interaction, the agent takes action a by policy π using the observed state s, then receives a feedback reward r and an updated state s′ from the environment. The agent aims to find an optimal policy to maximize the cumulative reward over the continuous interactions.The inference process of iterative algorithms can be described as MDP, as shown in Figure2. The agent policy is the model function π=f. At the time k, the state is sk=x,ϕk−1, the action is produced by the policy (namely the model function), and it is ak=ϕk. The reward is some negative distance function −dϕk,ϕ∗ where ϕ∗ denotes the optimum solution with respect to x. The environment is then transferred to the updated state sk+1=x,ϕk. In such formulation, the iterative inference is in consistent with the MDP.Figure 2
An illustrative interaction of an MDP.We can prove that: when the performance of the intermediate solution−dϕk,ϕ∗ is monotonously increasing, then the model f is a surjection and thus the data divergence does not exist.Proof.
Given any inputx and intermediate solution ϕi, ϕj,i,j∈0,1,…,Kmax,i≠j. When −dϕi,ϕ∗<−dϕj,ϕ∗, then according to the monotonicity of the distance function, we have ϕi≠ϕj.
Given a solution setϕkk=0Kmax where the elements satisfy −dϕ0,ϕ∗<⋯<−dϕKmax,ϕ∗, and thus we have ϕi≠ϕj,i,j∈0,1,…,Kmax,i≠j. The mapping of any input x,ϕk−1 is a unique ϕk. Therefore, the mapping f is a bijection and also a surjection, and the data divergence does not exist.The above conclusion is derived without the assumption of sufficient model complexity. When the monotonously increasing condition is not satisfied, the oscillation possibly occurs. Besides, when the monotonously increasing condition is satisfied, thedata divergence does not exist in the situation where the model parameter is not shared and the model complexity is limited.


## 3. Data-Driven Iterative Detection

### 3.1. System Model and Problem Formulation

As shown in Figure3, a point-to-point band-pass wireless communication system is considered. The symbol sequence vector s is produced with frame size NF, and they are modulated by the m-ary phase position shift keying (MPPSK) modulation [30]. The MPPSK signal of the symbol m∈1,…,M is given as(23)gmt=sin2πfct,0≤t<m−1KTc,−sin2πfct,m−1KTc≤t<mKTc,sin2πfct,mKTc≤t<NTc,where Tc=1/fc represents the carrier period, andK and N denote the number of carrier periods in each time slot and in each symbol, respectively. Apparently, we have N=KM and T=NTc, where T denotes the symbol duration. Under the MPPSK modulation, transmitted symbols are mapped into a base-band signal slt which can be written as(24)slt=∑n=1NFgnt−nT,where gnt∈gmtm=1M. Then, slt is up-converted by carrier frequency fc into a signal st=sltej2πfct, and shaped by a band-pass IIR filter with the impulse response hbpt, to be transmitted radio frequency signal xt. The additive white Gaussian noise (AWGN) channel is considered, and additional noise is denoted by wt with variance σw2. At the transmitter, a same band-pass IIR filter is used to denoise and obviate the disturbances. The filtered received signal zt is obtained as(25)zt=st∗hbpt+wt∗hbpt=s″t+w′t,where ∗ represents convolution operator. It can be seen that zt is composed of two parts: the colored Gaussian noise w′t which is filtered once, and the band-pass signal s″t which is filtered twice. The continuous signal zt is then sampled at sampling frequency fs where fs is usually a positive integral multiple of fc, i.e., fs/fc∈ℕ+, and finally we have sampled signal zup. In this work, we investigate the detection problem: estimate the transmitted symbol sequence s with the received sampled signal zup.Figure 3
An illustration of a band-pass wireless communication system.The band-pass IIR filters are deployed at both the transmitter and receiver to shape the transmitting signal and obviate the disturbances, respectively. However, the filters also introduce ISI, waveform distortions, and nonwhite noise. To overcome these issues, a matched filter, an equalizer, and a demodulator are usually required. The receiver design is intractable and thus we proposed an end-to-end data-driven approach.


### 3.2. Decision Feedback Receiver

In [25], we proposed an end-to-end NN-based receiver which iteratively estimates the transmitted symbols, and we called it decision feedback receiver (DFR). There are L hidden layers in the full-connected NN, and the number of neurons in the lth hidden layer is Nl. In the kth iteration, to detect symbol n, the input of DFR includes a windowed received sampled signal znup and a prior information vector qnk−1, and the output pnk is a conditional probability vector. The mapping of the DFR is expressed as(26)pnk=fznup,qnk−1;θ,where θ denotes the model parameter. As shown in Figure 4, znup is a vector filtered by a rectangular window. The window length is Ns×fs/fc where Ns∈ℕ+, and the window center is on the symbol n. The prior information vector qnk−1 is composed of the conditional probability vectors of the Npre∈ℕ+ previous and Npos∈ℕ+ posterior adjacent symbols of symbol n. Therefore, the number of feedback symbols is Nf=Npre+Npos, and the length of priori information vector is M×Nf. Then, the length of DFR input is given as(27)Nin=N×fsfc×Ns+M×Nf.Figure 4
An illustration of the received sampled signalznup and prior information vector qnk−1.Besides, the length of DFR output is equal toM. The estimated symbol can be achieved by(28)s^nk=argmaxm∈1,…,Mpnk.Summarily, the DFR utilizes the prior soft information of adjacent symbols derived from the last estimation, to iteratively estimate the transmitted symbols.


### 3.3. Existing Problems in DFR

Simulation results in [25] showed that after several iterations the DFR had better detection performance than the NN-based receiver without feedbacks. However, we also found that the first solution of DFR performed worse than the receiver without feedbacks, but theoretically they are the same. This phenomenon is caused by the mismatch between the training set and the test set. In (26), the DFR obtains the estimated soft information derived from the last estimation. However, the soft information q is unavailable in the training set, and q is replaced by the hard information which is generated from the correct output. From the analysis in subsection 2.1, we know that the mismatch in the input will lead to performance degradation.Owing to the mismatch in the input data between the training set and test set, three sub-problems arise and we summarize them as follows:(i)
First, according to the analysis in subsection2.1, the data missing exists in DFR due to the mismatch problem. When the band is narrow and the signal-to-noise ratio (SNR) is low, the mismatch becomes severe and the DFR detection performance on the test set is ruined.(ii)
Second, thedata divergence can exist due to the insufficient complexity of DFR. There is only one single model in (26), and its parameter θ is shared in all iterations.(iii)
Third, when the soft information of both posterior and previous adjacent symbols are utilized, the detection becomes unstable and oscillation occurs. Therefore, we adopt an alternative method where only the soft information of previous adjacent symbols is used as feedbacks.


### 3.4. Improved-DFR

#### 3.4.1. Framework of Improved-DFR

To overcome these existing problems, we unfold the DFR and make adjustments to the training algorithm via deep unfolding, and the improved-DFR is proposed. As shown in Figure5, the iterative algorithm is unfolded into Kmax sub-NNs, and the parameter of NN−k is θk, All the sub-NNs have the same structures and scale. The activation function in the hidden layers and the output layer are sigmoid and softmax, respectively. In DFR, the detection is frame-by-frame, and thus we use tensors to describe the data. The first dimension of tensor-like data refer to the sample number, which is the frame size NF in DFR. The tensor is a matrix X when the sample is a vector x. The mapping of sub-NN can be described as(29)Pk=fZup,Qk−1;θk,k∈1,…,Kmax.Figure 5
The illustrative framework of the improved-DFR.We useXk=Zup,Qk−1 to denote the input of NN−k. In the first iteration, Q0 is initialized as a zero matrix.In the proposed improved-DFR, Memory I transforms the received sampled signalzup into Zup and stores Zup. Besides, the output matrix Pk−1 is transmitted to Memory II−k and then transformed into priori information matrix Qk−1. The Qk−1 is stored in Memory II−k−1, and then transmitted to the NN−k as input. There are one Memory I and Kmax Memory IIs in the improved-DFR. In Algorithm 1, the detection algorithm of improved-DFR is given in the symbol-wise form. In addition, the detection algorithm can be parallel executed. The detailed hyper-parameters of the improved-DFR are given in Table1.Table 1
The hyper-parameters in the improved-DFR.
Hyper-parameterValue or settingSNR range0,10 dBNumber of input symbolsNs4Number of previous feedback symbolsNpre2Number of posterior feedback symbolsNpos1Model scale32Activation function in hidden layersSigmoid:fxi=1/1+exiActivation function in output layerSoftmax:fxi=exi/∑jexjOptimizerAdamInitial learning rateη1.0×10−3Number of framesNsam200Number of training episodesNepisode100Batch size (frame sizeNF)500Iteration timesKmax5Algorithm 1: The iterative detection algorithm of improved-DFR.
(1)
Input: The received sampled signal zup, frame size NF, iteration times Kmax, terminal threshold θth,(2)
Initialization: Transform zup to Zup and store Zup in Memory I. Initialize P0 as 0, initialize s^0 as 0,(3)
fork=1 to Kmaxdo(4)
TransformPk−1 to Qk−1, store Qk−1 in Memory II−k−1,(5)
forn=1 to NFdo(6)
xnk=znup,T,qnk−1,TT,(7)
pnk=fxnk;θk,(8)
s^nk=argmaxpnk,(9)
end for(10)
sk=s1k,…,sNFkT,(11)
ifs^k−s^k−10≤θththen(12)
break(13)
end if(14)
end for(15)
Output: s^=s^k


#### 3.4.2. Training of Improved-DFR

First, the received sampled signalzup and the corresponding transmitted symbol sequence s, respectively, are transformed into the input Zup and the output P. Given initial Q0 and input Zup, the improved-DFR serially obtains the solutions of each sub-NN, and we obtain the solution set Pkk=1Kmax. According to the cross-entropy loss function, the error of one frame is expressed as follows:(30)LP,Pkk=1Kmax=−1NF∑n=1NF∑k=1Kmax∑m=1Mpn,mlogpn,mk+1−pn,mlog1−pn,mk.To minimize (30), the parameter is updated along the negative gradient direction during learning:(31)θk←θk−η∇θkL,∀k∈1,…,Kmax,where η denotes the learning rate. Especially, we use the Adam optimizer [31–33] to dynamically adjust the learning rate. In Algorithm 2, the training algorithm is given in the parallel processing form. We summarize the improvements of improved-DFR as follows:(i)
During learning, the training set is dynamically generated where the priori information is transformed by the last solution of the previous sub-NN. Therefore, the training data and test data follow the same distribution, and the error caused bydata missing is eliminated.(ii)
Each sub-NN is trained with its parameter, and the possibly existingdata divergence caused by insufficient model complexity is avoided.(iii)
The soft information of both posterior and previous adjacent symbols are utilized as feedback, and the detection instability does not exist.Algorithm 2: The training algorithm of improved-DFR.
(1)
Input: Frame number Nsam, the received sampled signal set zuupu=1Nsam and the corresponding transmitted symbol sequence set suu=1Nsam, initial learning rate η, frame size NF, training episode times Nepisode, iteration times Kmax,(2)
Initialization: Randomly initialize θkk=1Kmax, initialize Adam with learning rate η,(3)
fori=1 to Nepisodedo(4)
forj=1 to Nsamdo(5)
Randomly selectzup in zuupu=1Nsam and the corresponding s in suu=1Nsam. Transform zup to Zup, transform s to P in 0−1 format(6)
InitializeP0 to be 0(7)
fork=1 to Kmaxdo(8)
TransformPk−1 to Qk−1,(9)
Xk=Zup,Qk−1,(10)
Pk=fXk;θk(11)
end for(12)
According to (30), calculate error LP,Pkk=1Kmax with P and Pkk=1Kmax(13)
According to (31), use Adam to calculate gradients ∇θkLk=1Kmax and update the parameter set θkk=1Kmax(14)
end for(15)
end for(16)
Output: θkk=1Kmax


#### 3.4.3. Practical Tricks in Training

In practical training, the numeric overflow problem often occurs, especially when the logarithmic calculation or index calculation is considered. Take the DFR as an example, the target output of the symbolm is a one-hot vector where the mth element is 1 and the other elements are 0. The m th output element of the model is(32)pm=eWmx+bm∑neWnx+bn=eWmx+bmeWmx+bm+∑n≠meWnx+bn=11+∑n≠meWnx+bn/eWmx+bm,where W and b are the parameters of the previous layer. When the model approximates the mth to be 1, from (32) we know that only when Wnx+bn⟶+∞, then pm⟶1. The parameters Wn, bn will sharply increase and thus the numeric overflow is caused. To fix this issue, one simple and efficient trick is to clip the element of output by the following function:(33)pm=maxpm,1−ϵ,m∈1,…,M.where ϵ is a small value, and ϵ=1×10−4 in our training.On the other hand, overfitting often exists in practical training, due to the unbalance between the model complexity and sample complexity. We use a validation set to observe the overfitting and evaluate the DFR performance after each training episode, and the model with the smallest error on the validation set is saved and then executed on the test set.


#### 3.4.4. Improved-DFR with Parameter Sharing

As we have discussed in subsection2.4, parameter sharing can have several benefits and it is still promising. The intensity of possible existing data divergence usually depends on the specific problem, and it can be alleviated by increasing the complexity of the single model. The execution and training algorithm of improved-DFR with parameter sharing can be regarded as one specific case of the improved-DFR without shared parameters. We change the parameter set θkk=1Kmax to be the shared parameter θ, and then the improved-DFR with parameter sharing is derived. In short, we call improved-DFR with parameter sharing as DFR-IS, and improved-DFR without parameter sharing as DFR-I.


#### 3.4.5. Complexity Analysis

First, we consider the computation complexity of receivers with various model scales. The computation complexity of the DFR and the improved-DFR is the same. In each inference per iteration and symbol, the addition calculation timesNadd, multiplication calculation times Nmul, and exponentiation calculation times Nexp are listed in Table 2. Both the computation complexity and time cost are proportion to the iteration times Kmax. Meanwhile, all the symbols in one frame can be parallel processed. Therefore, when the parallel computation is fully utilized, the time cost of inference is unchanged with various frame sizes.Table 2
Computation complexity per iteration and symbol.
Single hidden-layerDouble hidden-layerNaddNin×N1+N1×M−1Nin×N1+N1×N2+N2×M−1NmulNin+1×N1+N1+1×MNin+1×N1+N1+1×N2+N2+1×MNexpN1+MN1+N2+MNθNin+1×N1+N1+1×MNin+1×N1+N1+1×N2+N2+1×MSecond, we consider the model complexity and storage complexity of the improved-DFR. The number of parametersNθ and the amount of stored data Nmem are listed in Table 3. The stored data include: the received sampled signal zup and its transformation Zup in Memory I, the DFR output P, and its transformation Q in all Memory IIs. It is noteworthy that the Kmax Memory IIs are virtual, and in fact there is only one physical Memory II in DFRs.Table 3
Model complexity and storage complexity.
NθNmemDFR-INmul×Kmax2×Nin×NF+Nf+1×M×NFDFR-ISNmul2×Nin×NF+Nf+1×M×NF


## 3.1. System Model and Problem Formulation

As shown in Figure3, a point-to-point band-pass wireless communication system is considered. The symbol sequence vector s is produced with frame size NF, and they are modulated by the m-ary phase position shift keying (MPPSK) modulation [30]. The MPPSK signal of the symbol m∈1,…,M is given as(23)gmt=sin2πfct,0≤t<m−1KTc,−sin2πfct,m−1KTc≤t<mKTc,sin2πfct,mKTc≤t<NTc,where Tc=1/fc represents the carrier period, andK and N denote the number of carrier periods in each time slot and in each symbol, respectively. Apparently, we have N=KM and T=NTc, where T denotes the symbol duration. Under the MPPSK modulation, transmitted symbols are mapped into a base-band signal slt which can be written as(24)slt=∑n=1NFgnt−nT,where gnt∈gmtm=1M. Then, slt is up-converted by carrier frequency fc into a signal st=sltej2πfct, and shaped by a band-pass IIR filter with the impulse response hbpt, to be transmitted radio frequency signal xt. The additive white Gaussian noise (AWGN) channel is considered, and additional noise is denoted by wt with variance σw2. At the transmitter, a same band-pass IIR filter is used to denoise and obviate the disturbances. The filtered received signal zt is obtained as(25)zt=st∗hbpt+wt∗hbpt=s″t+w′t,where ∗ represents convolution operator. It can be seen that zt is composed of two parts: the colored Gaussian noise w′t which is filtered once, and the band-pass signal s″t which is filtered twice. The continuous signal zt is then sampled at sampling frequency fs where fs is usually a positive integral multiple of fc, i.e., fs/fc∈ℕ+, and finally we have sampled signal zup. In this work, we investigate the detection problem: estimate the transmitted symbol sequence s with the received sampled signal zup.Figure 3
An illustration of a band-pass wireless communication system.The band-pass IIR filters are deployed at both the transmitter and receiver to shape the transmitting signal and obviate the disturbances, respectively. However, the filters also introduce ISI, waveform distortions, and nonwhite noise. To overcome these issues, a matched filter, an equalizer, and a demodulator are usually required. The receiver design is intractable and thus we proposed an end-to-end data-driven approach.


## 3.2. Decision Feedback Receiver

In [25], we proposed an end-to-end NN-based receiver which iteratively estimates the transmitted symbols, and we called it decision feedback receiver (DFR). There are L hidden layers in the full-connected NN, and the number of neurons in the lth hidden layer is Nl. In the kth iteration, to detect symbol n, the input of DFR includes a windowed received sampled signal znup and a prior information vector qnk−1, and the output pnk is a conditional probability vector. The mapping of the DFR is expressed as(26)pnk=fznup,qnk−1;θ,where θ denotes the model parameter. As shown in Figure 4, znup is a vector filtered by a rectangular window. The window length is Ns×fs/fc where Ns∈ℕ+, and the window center is on the symbol n. The prior information vector qnk−1 is composed of the conditional probability vectors of the Npre∈ℕ+ previous and Npos∈ℕ+ posterior adjacent symbols of symbol n. Therefore, the number of feedback symbols is Nf=Npre+Npos, and the length of priori information vector is M×Nf. Then, the length of DFR input is given as(27)Nin=N×fsfc×Ns+M×Nf.Figure 4
An illustration of the received sampled signalznup and prior information vector qnk−1.Besides, the length of DFR output is equal toM. The estimated symbol can be achieved by(28)s^nk=argmaxm∈1,…,Mpnk.Summarily, the DFR utilizes the prior soft information of adjacent symbols derived from the last estimation, to iteratively estimate the transmitted symbols.


## 3.3. Existing Problems in DFR

Simulation results in [25] showed that after several iterations the DFR had better detection performance than the NN-based receiver without feedbacks. However, we also found that the first solution of DFR performed worse than the receiver without feedbacks, but theoretically they are the same. This phenomenon is caused by the mismatch between the training set and the test set. In (26), the DFR obtains the estimated soft information derived from the last estimation. However, the soft information q is unavailable in the training set, and q is replaced by the hard information which is generated from the correct output. From the analysis in subsection 2.1, we know that the mismatch in the input will lead to performance degradation.Owing to the mismatch in the input data between the training set and test set, three sub-problems arise and we summarize them as follows:(i)
First, according to the analysis in subsection2.1, the data missing exists in DFR due to the mismatch problem. When the band is narrow and the signal-to-noise ratio (SNR) is low, the mismatch becomes severe and the DFR detection performance on the test set is ruined.(ii)
Second, thedata divergence can exist due to the insufficient complexity of DFR. There is only one single model in (26), and its parameter θ is shared in all iterations.(iii)
Third, when the soft information of both posterior and previous adjacent symbols are utilized, the detection becomes unstable and oscillation occurs. Therefore, we adopt an alternative method where only the soft information of previous adjacent symbols is used as feedbacks.


## 3.4. Improved-DFR

### 3.4.1. Framework of Improved-DFR

To overcome these existing problems, we unfold the DFR and make adjustments to the training algorithm via deep unfolding, and the improved-DFR is proposed. As shown in Figure5, the iterative algorithm is unfolded into Kmax sub-NNs, and the parameter of NN−k is θk, All the sub-NNs have the same structures and scale. The activation function in the hidden layers and the output layer are sigmoid and softmax, respectively. In DFR, the detection is frame-by-frame, and thus we use tensors to describe the data. The first dimension of tensor-like data refer to the sample number, which is the frame size NF in DFR. The tensor is a matrix X when the sample is a vector x. The mapping of sub-NN can be described as(29)Pk=fZup,Qk−1;θk,k∈1,…,Kmax.Figure 5
The illustrative framework of the improved-DFR.We useXk=Zup,Qk−1 to denote the input of NN−k. In the first iteration, Q0 is initialized as a zero matrix.In the proposed improved-DFR, Memory I transforms the received sampled signalzup into Zup and stores Zup. Besides, the output matrix Pk−1 is transmitted to Memory II−k and then transformed into priori information matrix Qk−1. The Qk−1 is stored in Memory II−k−1, and then transmitted to the NN−k as input. There are one Memory I and Kmax Memory IIs in the improved-DFR. In Algorithm 1, the detection algorithm of improved-DFR is given in the symbol-wise form. In addition, the detection algorithm can be parallel executed. The detailed hyper-parameters of the improved-DFR are given in Table1.Table 1
The hyper-parameters in the improved-DFR.
Hyper-parameterValue or settingSNR range0,10 dBNumber of input symbolsNs4Number of previous feedback symbolsNpre2Number of posterior feedback symbolsNpos1Model scale32Activation function in hidden layersSigmoid:fxi=1/1+exiActivation function in output layerSoftmax:fxi=exi/∑jexjOptimizerAdamInitial learning rateη1.0×10−3Number of framesNsam200Number of training episodesNepisode100Batch size (frame sizeNF)500Iteration timesKmax5Algorithm 1: The iterative detection algorithm of improved-DFR.
(1)
Input: The received sampled signal zup, frame size NF, iteration times Kmax, terminal threshold θth,(2)
Initialization: Transform zup to Zup and store Zup in Memory I. Initialize P0 as 0, initialize s^0 as 0,(3)
fork=1 to Kmaxdo(4)
TransformPk−1 to Qk−1, store Qk−1 in Memory II−k−1,(5)
forn=1 to NFdo(6)
xnk=znup,T,qnk−1,TT,(7)
pnk=fxnk;θk,(8)
s^nk=argmaxpnk,(9)
end for(10)
sk=s1k,…,sNFkT,(11)
ifs^k−s^k−10≤θththen(12)
break(13)
end if(14)
end for(15)
Output: s^=s^k


### 3.4.2. Training of Improved-DFR

First, the received sampled signalzup and the corresponding transmitted symbol sequence s, respectively, are transformed into the input Zup and the output P. Given initial Q0 and input Zup, the improved-DFR serially obtains the solutions of each sub-NN, and we obtain the solution set Pkk=1Kmax. According to the cross-entropy loss function, the error of one frame is expressed as follows:(30)LP,Pkk=1Kmax=−1NF∑n=1NF∑k=1Kmax∑m=1Mpn,mlogpn,mk+1−pn,mlog1−pn,mk.To minimize (30), the parameter is updated along the negative gradient direction during learning:(31)θk←θk−η∇θkL,∀k∈1,…,Kmax,where η denotes the learning rate. Especially, we use the Adam optimizer [31–33] to dynamically adjust the learning rate. In Algorithm 2, the training algorithm is given in the parallel processing form. We summarize the improvements of improved-DFR as follows:(i)
During learning, the training set is dynamically generated where the priori information is transformed by the last solution of the previous sub-NN. Therefore, the training data and test data follow the same distribution, and the error caused bydata missing is eliminated.(ii)
Each sub-NN is trained with its parameter, and the possibly existingdata divergence caused by insufficient model complexity is avoided.(iii)
The soft information of both posterior and previous adjacent symbols are utilized as feedback, and the detection instability does not exist.Algorithm 2: The training algorithm of improved-DFR.
(1)
Input: Frame number Nsam, the received sampled signal set zuupu=1Nsam and the corresponding transmitted symbol sequence set suu=1Nsam, initial learning rate η, frame size NF, training episode times Nepisode, iteration times Kmax,(2)
Initialization: Randomly initialize θkk=1Kmax, initialize Adam with learning rate η,(3)
fori=1 to Nepisodedo(4)
forj=1 to Nsamdo(5)
Randomly selectzup in zuupu=1Nsam and the corresponding s in suu=1Nsam. Transform zup to Zup, transform s to P in 0−1 format(6)
InitializeP0 to be 0(7)
fork=1 to Kmaxdo(8)
TransformPk−1 to Qk−1,(9)
Xk=Zup,Qk−1,(10)
Pk=fXk;θk(11)
end for(12)
According to (30), calculate error LP,Pkk=1Kmax with P and Pkk=1Kmax(13)
According to (31), use Adam to calculate gradients ∇θkLk=1Kmax and update the parameter set θkk=1Kmax(14)
end for(15)
end for(16)
Output: θkk=1Kmax


### 3.4.3. Practical Tricks in Training

In practical training, the numeric overflow problem often occurs, especially when the logarithmic calculation or index calculation is considered. Take the DFR as an example, the target output of the symbolm is a one-hot vector where the mth element is 1 and the other elements are 0. The m th output element of the model is(32)pm=eWmx+bm∑neWnx+bn=eWmx+bmeWmx+bm+∑n≠meWnx+bn=11+∑n≠meWnx+bn/eWmx+bm,where W and b are the parameters of the previous layer. When the model approximates the mth to be 1, from (32) we know that only when Wnx+bn⟶+∞, then pm⟶1. The parameters Wn, bn will sharply increase and thus the numeric overflow is caused. To fix this issue, one simple and efficient trick is to clip the element of output by the following function:(33)pm=maxpm,1−ϵ,m∈1,…,M.where ϵ is a small value, and ϵ=1×10−4 in our training.On the other hand, overfitting often exists in practical training, due to the unbalance between the model complexity and sample complexity. We use a validation set to observe the overfitting and evaluate the DFR performance after each training episode, and the model with the smallest error on the validation set is saved and then executed on the test set.


### 3.4.4. Improved-DFR with Parameter Sharing

As we have discussed in subsection2.4, parameter sharing can have several benefits and it is still promising. The intensity of possible existing data divergence usually depends on the specific problem, and it can be alleviated by increasing the complexity of the single model. The execution and training algorithm of improved-DFR with parameter sharing can be regarded as one specific case of the improved-DFR without shared parameters. We change the parameter set θkk=1Kmax to be the shared parameter θ, and then the improved-DFR with parameter sharing is derived. In short, we call improved-DFR with parameter sharing as DFR-IS, and improved-DFR without parameter sharing as DFR-I.


### 3.4.5. Complexity Analysis

First, we consider the computation complexity of receivers with various model scales. The computation complexity of the DFR and the improved-DFR is the same. In each inference per iteration and symbol, the addition calculation timesNadd, multiplication calculation times Nmul, and exponentiation calculation times Nexp are listed in Table 2. Both the computation complexity and time cost are proportion to the iteration times Kmax. Meanwhile, all the symbols in one frame can be parallel processed. Therefore, when the parallel computation is fully utilized, the time cost of inference is unchanged with various frame sizes.Table 2
Computation complexity per iteration and symbol.
Single hidden-layerDouble hidden-layerNaddNin×N1+N1×M−1Nin×N1+N1×N2+N2×M−1NmulNin+1×N1+N1+1×MNin+1×N1+N1+1×N2+N2+1×MNexpN1+MN1+N2+MNθNin+1×N1+N1+1×MNin+1×N1+N1+1×N2+N2+1×MSecond, we consider the model complexity and storage complexity of the improved-DFR. The number of parametersNθ and the amount of stored data Nmem are listed in Table 3. The stored data include: the received sampled signal zup and its transformation Zup in Memory I, the DFR output P, and its transformation Q in all Memory IIs. It is noteworthy that the Kmax Memory IIs are virtual, and in fact there is only one physical Memory II in DFRs.Table 3
Model complexity and storage complexity.
NθNmemDFR-INmul×Kmax2×Nin×NF+Nf+1×M×NFDFR-ISNmul2×Nin×NF+Nf+1×M×NF


## 3.4.1. Framework of Improved-DFR

To overcome these existing problems, we unfold the DFR and make adjustments to the training algorithm via deep unfolding, and the improved-DFR is proposed. As shown in Figure5, the iterative algorithm is unfolded into Kmax sub-NNs, and the parameter of NN−k is θk, All the sub-NNs have the same structures and scale. The activation function in the hidden layers and the output layer are sigmoid and softmax, respectively. In DFR, the detection is frame-by-frame, and thus we use tensors to describe the data. The first dimension of tensor-like data refer to the sample number, which is the frame size NF in DFR. The tensor is a matrix X when the sample is a vector x. The mapping of sub-NN can be described as(29)Pk=fZup,Qk−1;θk,k∈1,…,Kmax.Figure 5
The illustrative framework of the improved-DFR.We useXk=Zup,Qk−1 to denote the input of NN−k. In the first iteration, Q0 is initialized as a zero matrix.In the proposed improved-DFR, Memory I transforms the received sampled signalzup into Zup and stores Zup. Besides, the output matrix Pk−1 is transmitted to Memory II−k and then transformed into priori information matrix Qk−1. The Qk−1 is stored in Memory II−k−1, and then transmitted to the NN−k as input. There are one Memory I and Kmax Memory IIs in the improved-DFR. In Algorithm 1, the detection algorithm of improved-DFR is given in the symbol-wise form. In addition, the detection algorithm can be parallel executed. The detailed hyper-parameters of the improved-DFR are given in Table1.Table 1
The hyper-parameters in the improved-DFR.
Hyper-parameterValue or settingSNR range0,10 dBNumber of input symbolsNs4Number of previous feedback symbolsNpre2Number of posterior feedback symbolsNpos1Model scale32Activation function in hidden layersSigmoid:fxi=1/1+exiActivation function in output layerSoftmax:fxi=exi/∑jexjOptimizerAdamInitial learning rateη1.0×10−3Number of framesNsam200Number of training episodesNepisode100Batch size (frame sizeNF)500Iteration timesKmax5Algorithm 1: The iterative detection algorithm of improved-DFR.
(1)
Input: The received sampled signal zup, frame size NF, iteration times Kmax, terminal threshold θth,(2)
Initialization: Transform zup to Zup and store Zup in Memory I. Initialize P0 as 0, initialize s^0 as 0,(3)
fork=1 to Kmaxdo(4)
TransformPk−1 to Qk−1, store Qk−1 in Memory II−k−1,(5)
forn=1 to NFdo(6)
xnk=znup,T,qnk−1,TT,(7)
pnk=fxnk;θk,(8)
s^nk=argmaxpnk,(9)
end for(10)
sk=s1k,…,sNFkT,(11)
ifs^k−s^k−10≤θththen(12)
break(13)
end if(14)
end for(15)
Output: s^=s^k


## 3.4.2. Training of Improved-DFR

First, the received sampled signalzup and the corresponding transmitted symbol sequence s, respectively, are transformed into the input Zup and the output P. Given initial Q0 and input Zup, the improved-DFR serially obtains the solutions of each sub-NN, and we obtain the solution set Pkk=1Kmax. According to the cross-entropy loss function, the error of one frame is expressed as follows:(30)LP,Pkk=1Kmax=−1NF∑n=1NF∑k=1Kmax∑m=1Mpn,mlogpn,mk+1−pn,mlog1−pn,mk.To minimize (30), the parameter is updated along the negative gradient direction during learning:(31)θk←θk−η∇θkL,∀k∈1,…,Kmax,where η denotes the learning rate. Especially, we use the Adam optimizer [31–33] to dynamically adjust the learning rate. In Algorithm 2, the training algorithm is given in the parallel processing form. We summarize the improvements of improved-DFR as follows:(i)
During learning, the training set is dynamically generated where the priori information is transformed by the last solution of the previous sub-NN. Therefore, the training data and test data follow the same distribution, and the error caused bydata missing is eliminated.(ii)
Each sub-NN is trained with its parameter, and the possibly existingdata divergence caused by insufficient model complexity is avoided.(iii)
The soft information of both posterior and previous adjacent symbols are utilized as feedback, and the detection instability does not exist.Algorithm 2: The training algorithm of improved-DFR.
(1)
Input: Frame number Nsam, the received sampled signal set zuupu=1Nsam and the corresponding transmitted symbol sequence set suu=1Nsam, initial learning rate η, frame size NF, training episode times Nepisode, iteration times Kmax,(2)
Initialization: Randomly initialize θkk=1Kmax, initialize Adam with learning rate η,(3)
fori=1 to Nepisodedo(4)
forj=1 to Nsamdo(5)
Randomly selectzup in zuupu=1Nsam and the corresponding s in suu=1Nsam. Transform zup to Zup, transform s to P in 0−1 format(6)
InitializeP0 to be 0(7)
fork=1 to Kmaxdo(8)
TransformPk−1 to Qk−1,(9)
Xk=Zup,Qk−1,(10)
Pk=fXk;θk(11)
end for(12)
According to (30), calculate error LP,Pkk=1Kmax with P and Pkk=1Kmax(13)
According to (31), use Adam to calculate gradients ∇θkLk=1Kmax and update the parameter set θkk=1Kmax(14)
end for(15)
end for(16)
Output: θkk=1Kmax


## 3.4.3. Practical Tricks in Training

In practical training, the numeric overflow problem often occurs, especially when the logarithmic calculation or index calculation is considered. Take the DFR as an example, the target output of the symbolm is a one-hot vector where the mth element is 1 and the other elements are 0. The m th output element of the model is(32)pm=eWmx+bm∑neWnx+bn=eWmx+bmeWmx+bm+∑n≠meWnx+bn=11+∑n≠meWnx+bn/eWmx+bm,where W and b are the parameters of the previous layer. When the model approximates the mth to be 1, from (32) we know that only when Wnx+bn⟶+∞, then pm⟶1. The parameters Wn, bn will sharply increase and thus the numeric overflow is caused. To fix this issue, one simple and efficient trick is to clip the element of output by the following function:(33)pm=maxpm,1−ϵ,m∈1,…,M.where ϵ is a small value, and ϵ=1×10−4 in our training.On the other hand, overfitting often exists in practical training, due to the unbalance between the model complexity and sample complexity. We use a validation set to observe the overfitting and evaluate the DFR performance after each training episode, and the model with the smallest error on the validation set is saved and then executed on the test set.


## 3.4.4. Improved-DFR with Parameter Sharing

As we have discussed in subsection2.4, parameter sharing can have several benefits and it is still promising. The intensity of possible existing data divergence usually depends on the specific problem, and it can be alleviated by increasing the complexity of the single model. The execution and training algorithm of improved-DFR with parameter sharing can be regarded as one specific case of the improved-DFR without shared parameters. We change the parameter set θkk=1Kmax to be the shared parameter θ, and then the improved-DFR with parameter sharing is derived. In short, we call improved-DFR with parameter sharing as DFR-IS, and improved-DFR without parameter sharing as DFR-I.


## 3.4.5. Complexity Analysis

First, we consider the computation complexity of receivers with various model scales. The computation complexity of the DFR and the improved-DFR is the same. In each inference per iteration and symbol, the addition calculation timesNadd, multiplication calculation times Nmul, and exponentiation calculation times Nexp are listed in Table 2. Both the computation complexity and time cost are proportion to the iteration times Kmax. Meanwhile, all the symbols in one frame can be parallel processed. Therefore, when the parallel computation is fully utilized, the time cost of inference is unchanged with various frame sizes.Table 2
Computation complexity per iteration and symbol.
Single hidden-layerDouble hidden-layerNaddNin×N1+N1×M−1Nin×N1+N1×N2+N2×M−1NmulNin+1×N1+N1+1×MNin+1×N1+N1+1×N2+N2+1×MNexpN1+MN1+N2+MNθNin+1×N1+N1+1×MNin+1×N1+N1+1×N2+N2+1×MSecond, we consider the model complexity and storage complexity of the improved-DFR. The number of parametersNθ and the amount of stored data Nmem are listed in Table 3. The stored data include: the received sampled signal zup and its transformation Zup in Memory I, the DFR output P, and its transformation Q in all Memory IIs. It is noteworthy that the Kmax Memory IIs are virtual, and in fact there is only one physical Memory II in DFRs.Table 3
Model complexity and storage complexity.
NθNmemDFR-INmul×Kmax2×Nin×NF+Nf+1×M×NFDFR-ISNmul2×Nin×NF+Nf+1×M×NF


## 4. Simulation Results

The parameters of the band-pass wireless communication system are listed in Table4 With the values and settings in Table 4, the number of carrier periods in each symbol is N=8, and the order of the designed band-pass filter is 21. Besides, the detailed hyper-parameters of the improved-DFR are listed in Table 1. The number of feedback symbols is Nf=3, the length of priori information vector is 12, the length of input received sampled signal is 128, and the input vector length of DFR is Nin=140. The number of training samples is 1×105. The DFR-I and DFR-IS are tested in this section, and the DFR with Npre=3 and Npos=0 is regarded as a comparison. These values or settings are fixed in the following simulations and will be particularly mentioned if modified. The simulation results are averaged with 20 repeats.Table 4
The parameters of the band-pass wireless communication system.
ParameterValue or settingFrame sizeNF500Modulation levelM4Number of carrier periods in each time slotK2Carrier frequencyfc70 MHzSampling frequencyfs280 MHzBand-pass bandwidthB20 MHzBand-pass filter typeChebyshev IIMaximum band-pass rippleRp3 dBMinimum stop-band attenuationRs45 dBFrequency intervalΔf2 MHz


### 4.1. Model Complexity

First, we study the influences of the NN scale on the improved-DFR performance. The NNs having a single hidden layer with2,8,32,128 neurons, and double hidden layers with 64,64, 128,128 are considered. The single layer of 2 neurons is the possible smallest NN, and we assume that a NN with scale 128,128 has sufficient model complexity. We use the error function (33) to measure the training quality.


#### 4.1.1. Error

As shown in Figure6, the error on the training set is recorded along the episode times Nepisode. In general, the final error decreases as the model complexity increases, and the DFR-I has a smaller final training error than DFR-IS. Besides, the NNs of more complexities have faster convergence. These simulation results basically agree with the theoretical expectations. However, the final error of NN with 128,128 is larger than that with 64,64, and the DFR-I has a larger final training error than DFR-IS when the NNs are 128, 64,64, 128,128. We speculate that these abnormal phenomena are caused by the overfitting where the model complexity is excessively high.Figure 6
The error on training set versus episode times.Meanwhile, the error on the validation set is recorded in Figure7. As the episode times increase, the validation errors of NNs with 128, 64,64, and128,128 increase to varying degrees, which verifies our speculation that the overfitting occurs. The DFR-IS with 128 obtains the lowest final validation error. The error on the validation set also indicates that parameter sharing is useful to alleviate the overfitting when the model is complex.Figure 7
The error on validation set versus episode times.


#### 4.1.2. Detection

The learned models are then tested on the test set. After5 iteration times, the SER curves are plotted in Figure 8. Generally, the SNR–SER performance of DFR-I and DFR-IS is similar, but the ones using parameter sharing are slightly worse. All the SERs of improved-DFRs are lower than 1.0×10−5 when SNR=8 dB, except for the NN with 2. However, the smallest NN still can achieve SER=1.0×10−5 when the SNR is increased up to 10 dB.Figure 8
The SNR-SER curves of the improved-DFRs with various model scales.


#### 4.1.3. Complexity

The high-speed processing is significant for wireless communication systems, and thus the low complexity is preferred. The specific computation times and parameter number of different NNs are listed in Table5. Generally, the computation times are proportion to the sum amount of hidden neurons. Although the improved-DFR with NN being 2 suffers about 2 dB SNR loss to achieve SER=1.0×10−5, the corresponding computation cost and parameter number are extremely small. Taking account of training difficulty, detection performance, computation, and model complexity, an improved-DFR with 32 is adopted in the following simulations.Table 5
Computation times and parameter number.
NmulNaddNexpNθDFR−IS22872946287811511164121151324607464436460712818431185641321843164,64133111279413213311128,128348153507626034815


### 4.2. Iteration Times

In this subsection, we study the relationship between the iteration times and the detection performance, andSNR∈0,10 dB. As shown in Figure 9, the SNR–SER curves of DFR and DFR-I after 5 iterations are plotted. Generally, the SER of the DFR-I sharply declines in the first 3 iterations, then the SER almost no longer declines and thus the corresponding solution converges. Meanwhile, the SER of the DFR still slowly declines after 4 iterations, and the corresponding solution slowly converges. In the high SNR region, the SER gap between the DFR-I and the DFR becomes larger than that in the low SNR region. The SER performance of DFR after 3 iterations is comparable to that of DFR-I after the first iteration. This indicates that the mismatch between the training set and test set seriously ruins the performance of DFR. To achieve SER=1.0×10−5 after 5 iterations, the required SNR of the DFR is 10 dB, while the DFR-I only needs about SNR=7 dB.Figure 9
The SNR-SER curves of the DFR and the DFR-I with different iteration times.In summary, the improved-DFR addresses the mismatch problem between the training set and the testing set in the vanilla DFR, and thus the improved-DFR outperforms the DFR. In comparison to the DFR, the detection SER of the initial solution of the improved-DFR is greatly reduced. Besides, the convergence speed and the final SER of the improved-DFR respectively are faster and lower.On the other hand, we study the influences of parameter sharing on the detection performance of improved-DFR after different iteration times. As shown in Figure10, generally the SNR–SER performance of the DFR-I is slightly better than that of the DFR-IS which uses parameter sharing. When SNR=10 dB, symbol errors cannot be captured on the test set with the DFR-I after 3 iterations; meanwhile, the SER of the DFR-IS is around 1.0×10−6. To improve the detection performance of the improved-DFR using parameter sharing, we can properly increase the model complexity.Figure 10
The SNR-SER curves of the DFR-IS and the DFR-I after different iteration times.


### 4.3. Band-Pass Bandwidth

In this subsection, the DFR, DFR-I, and DFR-IS respectively are trained and tested in the scenarios with different band-pass bandwidthB. The bandwidth set is 15,20,25 MHz, and the elements respectively represent the corresponding scenarios with severe, normal, and slight ISI. Under these three different bandwidths, the SNR ranges of the corresponding test sets respectively are −2,8 dB, 0,12 dB, and 4,14 dB. Meanwhile, the SNR ranges of the training sets are all 0,10 dB.The SNR–SER curves of DFR-I after first iteration, and DFR, DFR-I, DFR-IS after5 iterations are illustrated in Figure 11. As a baseline, the DFR-I after first iteration is regarded as a normal NN-based receiver without feedbacks. In general, under different bandwidths, the DFR-I outperforms the other receivers, and the detection performance of improved-DFRs is better than the DFR. When the bandwidth is reduced to B=15 MHz, the ISI is serious, and the SERs of all receivers drop slowly. When SNR=14 dB, the estimated result of the normal receiver without feedbacks is SER=7.6×10−2. The performance improvement of DFR is poor, and its SER is only 1.6×10−2. In contrast, the SER of DFR-I and DFR-IS after 5 iterations are 1.5×10−3 and 2.1×10−3, respectively. Compared with DFR, the SER of improved-DFR is reduced by about one order of magnitude.Figure 11
The SNR-SER curves of DFR, DFR-IS, and DFR-I with different bandwidths.Then, we turn to the scenario withB=25 MHz. It can be seen that as the bandwidth increases, the ISI is alleviated. The detection performance of the three receivers gets closer, and the performance gain achieved by iterations is insignificant. However, the DFR-I still outperforms the other receivers. When B=25 MHz and SNR=6 dB, the SER of DFR-I is 2.5×10−6, while SER of the DFR-IS and DFR is around 1.0×10−5.


## 4.1. Model Complexity

First, we study the influences of the NN scale on the improved-DFR performance. The NNs having a single hidden layer with2,8,32,128 neurons, and double hidden layers with 64,64, 128,128 are considered. The single layer of 2 neurons is the possible smallest NN, and we assume that a NN with scale 128,128 has sufficient model complexity. We use the error function (33) to measure the training quality.


### 4.1.1. Error

As shown in Figure6, the error on the training set is recorded along the episode times Nepisode. In general, the final error decreases as the model complexity increases, and the DFR-I has a smaller final training error than DFR-IS. Besides, the NNs of more complexities have faster convergence. These simulation results basically agree with the theoretical expectations. However, the final error of NN with 128,128 is larger than that with 64,64, and the DFR-I has a larger final training error than DFR-IS when the NNs are 128, 64,64, 128,128. We speculate that these abnormal phenomena are caused by the overfitting where the model complexity is excessively high.Figure 6
The error on training set versus episode times.Meanwhile, the error on the validation set is recorded in Figure7. As the episode times increase, the validation errors of NNs with 128, 64,64, and128,128 increase to varying degrees, which verifies our speculation that the overfitting occurs. The DFR-IS with 128 obtains the lowest final validation error. The error on the validation set also indicates that parameter sharing is useful to alleviate the overfitting when the model is complex.Figure 7
The error on validation set versus episode times.


### 4.1.2. Detection

The learned models are then tested on the test set. After5 iteration times, the SER curves are plotted in Figure 8. Generally, the SNR–SER performance of DFR-I and DFR-IS is similar, but the ones using parameter sharing are slightly worse. All the SERs of improved-DFRs are lower than 1.0×10−5 when SNR=8 dB, except for the NN with 2. However, the smallest NN still can achieve SER=1.0×10−5 when the SNR is increased up to 10 dB.Figure 8
The SNR-SER curves of the improved-DFRs with various model scales.


### 4.1.3. Complexity

The high-speed processing is significant for wireless communication systems, and thus the low complexity is preferred. The specific computation times and parameter number of different NNs are listed in Table5. Generally, the computation times are proportion to the sum amount of hidden neurons. Although the improved-DFR with NN being 2 suffers about 2 dB SNR loss to achieve SER=1.0×10−5, the corresponding computation cost and parameter number are extremely small. Taking account of training difficulty, detection performance, computation, and model complexity, an improved-DFR with 32 is adopted in the following simulations.Table 5
Computation times and parameter number.
NmulNaddNexpNθDFR−IS22872946287811511164121151324607464436460712818431185641321843164,64133111279413213311128,128348153507626034815


## 4.1.1. Error

As shown in Figure6, the error on the training set is recorded along the episode times Nepisode. In general, the final error decreases as the model complexity increases, and the DFR-I has a smaller final training error than DFR-IS. Besides, the NNs of more complexities have faster convergence. These simulation results basically agree with the theoretical expectations. However, the final error of NN with 128,128 is larger than that with 64,64, and the DFR-I has a larger final training error than DFR-IS when the NNs are 128, 64,64, 128,128. We speculate that these abnormal phenomena are caused by the overfitting where the model complexity is excessively high.Figure 6
The error on training set versus episode times.Meanwhile, the error on the validation set is recorded in Figure7. As the episode times increase, the validation errors of NNs with 128, 64,64, and128,128 increase to varying degrees, which verifies our speculation that the overfitting occurs. The DFR-IS with 128 obtains the lowest final validation error. The error on the validation set also indicates that parameter sharing is useful to alleviate the overfitting when the model is complex.Figure 7
The error on validation set versus episode times.


## 4.1.2. Detection

The learned models are then tested on the test set. After5 iteration times, the SER curves are plotted in Figure 8. Generally, the SNR–SER performance of DFR-I and DFR-IS is similar, but the ones using parameter sharing are slightly worse. All the SERs of improved-DFRs are lower than 1.0×10−5 when SNR=8 dB, except for the NN with 2. However, the smallest NN still can achieve SER=1.0×10−5 when the SNR is increased up to 10 dB.Figure 8
The SNR-SER curves of the improved-DFRs with various model scales.


## 4.1.3. Complexity

The high-speed processing is significant for wireless communication systems, and thus the low complexity is preferred. The specific computation times and parameter number of different NNs are listed in Table5. Generally, the computation times are proportion to the sum amount of hidden neurons. Although the improved-DFR with NN being 2 suffers about 2 dB SNR loss to achieve SER=1.0×10−5, the corresponding computation cost and parameter number are extremely small. Taking account of training difficulty, detection performance, computation, and model complexity, an improved-DFR with 32 is adopted in the following simulations.Table 5
Computation times and parameter number.
NmulNaddNexpNθDFR−IS22872946287811511164121151324607464436460712818431185641321843164,64133111279413213311128,128348153507626034815


## 4.2. Iteration Times

In this subsection, we study the relationship between the iteration times and the detection performance, andSNR∈0,10 dB. As shown in Figure 9, the SNR–SER curves of DFR and DFR-I after 5 iterations are plotted. Generally, the SER of the DFR-I sharply declines in the first 3 iterations, then the SER almost no longer declines and thus the corresponding solution converges. Meanwhile, the SER of the DFR still slowly declines after 4 iterations, and the corresponding solution slowly converges. In the high SNR region, the SER gap between the DFR-I and the DFR becomes larger than that in the low SNR region. The SER performance of DFR after 3 iterations is comparable to that of DFR-I after the first iteration. This indicates that the mismatch between the training set and test set seriously ruins the performance of DFR. To achieve SER=1.0×10−5 after 5 iterations, the required SNR of the DFR is 10 dB, while the DFR-I only needs about SNR=7 dB.Figure 9
The SNR-SER curves of the DFR and the DFR-I with different iteration times.In summary, the improved-DFR addresses the mismatch problem between the training set and the testing set in the vanilla DFR, and thus the improved-DFR outperforms the DFR. In comparison to the DFR, the detection SER of the initial solution of the improved-DFR is greatly reduced. Besides, the convergence speed and the final SER of the improved-DFR respectively are faster and lower.On the other hand, we study the influences of parameter sharing on the detection performance of improved-DFR after different iteration times. As shown in Figure10, generally the SNR–SER performance of the DFR-I is slightly better than that of the DFR-IS which uses parameter sharing. When SNR=10 dB, symbol errors cannot be captured on the test set with the DFR-I after 3 iterations; meanwhile, the SER of the DFR-IS is around 1.0×10−6. To improve the detection performance of the improved-DFR using parameter sharing, we can properly increase the model complexity.Figure 10
The SNR-SER curves of the DFR-IS and the DFR-I after different iteration times.


## 4.3. Band-Pass Bandwidth

In this subsection, the DFR, DFR-I, and DFR-IS respectively are trained and tested in the scenarios with different band-pass bandwidthB. The bandwidth set is 15,20,25 MHz, and the elements respectively represent the corresponding scenarios with severe, normal, and slight ISI. Under these three different bandwidths, the SNR ranges of the corresponding test sets respectively are −2,8 dB, 0,12 dB, and 4,14 dB. Meanwhile, the SNR ranges of the training sets are all 0,10 dB.The SNR–SER curves of DFR-I after first iteration, and DFR, DFR-I, DFR-IS after5 iterations are illustrated in Figure 11. As a baseline, the DFR-I after first iteration is regarded as a normal NN-based receiver without feedbacks. In general, under different bandwidths, the DFR-I outperforms the other receivers, and the detection performance of improved-DFRs is better than the DFR. When the bandwidth is reduced to B=15 MHz, the ISI is serious, and the SERs of all receivers drop slowly. When SNR=14 dB, the estimated result of the normal receiver without feedbacks is SER=7.6×10−2. The performance improvement of DFR is poor, and its SER is only 1.6×10−2. In contrast, the SER of DFR-I and DFR-IS after 5 iterations are 1.5×10−3 and 2.1×10−3, respectively. Compared with DFR, the SER of improved-DFR is reduced by about one order of magnitude.Figure 11
The SNR-SER curves of DFR, DFR-IS, and DFR-I with different bandwidths.Then, we turn to the scenario withB=25 MHz. It can be seen that as the bandwidth increases, the ISI is alleviated. The detection performance of the three receivers gets closer, and the performance gain achieved by iterations is insignificant. However, the DFR-I still outperforms the other receivers. When B=25 MHz and SNR=6 dB, the SER of DFR-I is 2.5×10−6, while SER of the DFR-IS and DFR is around 1.0×10−5.


## 5. Conclusions

In a general way, we have quantitatively analyzed the increased test error brought by the mismatch problem between the training set and the test set, and the corresponding upper bound and lower bound are also given. We have pointed out that the increased test error is composed ofdata divergence and data missing, and the solution to eliminate the data missing and the conditions that the data divergence no longer exists are also given. The established analysis is further developed for the case where one single model is trained on an integral training set. We have proved that the deep unfolding using parameter sharing has no data divergence with sufficient model complexity. Meanwhile, with the aid of the MDP model, we have given the condition that the data divergence does not exist, when the model complexity is insufficient. Based on the aforementioned analysis, we have studied the mismatch problem and the caused sub-problems in the DFR. Then, the improvements to solve these problems are proposed and thus the improved-DFR is developed. The improved-DFR has low computation complexity and model complexity and can be executed by parallel processing. The simulation results show that the improved-DFR has a faster convergence speed and better final SER performance than the DFR. Moreover, the performance of DFR-I and DFR-IS is similar. In comparison to DFR-I, the DFR-IS is easier to train, and the slight performance loss can be reduced by increasing model complexity. In our future work, we will focus on the model and data-driven method for DFR, to further reduce the training complexity of the data-driven DFR.



---

*Source: 5656880-2022-08-24.xml*