# An Overview of FIR Filter Design in Future Multicarrier Communication Systems
## 3 FIR Prototype Filter Design
### Optimization Based Methods
#### 3.3.5 Karaboga's Method  
In Reference [135], an FIR filter is designed based on the differential evolution (DE) algorithm. The main difference between DE algorithm and GA algorithm is that GA algorithm relies on the crossover, while DE algorithm relies on the mutation operation. The specific steps of the DE algorithm are provided in Algorithm 2. In order to evaluate the performance of the designed FIR filters by GA and DE algorithms, the least mean squared error (LMSE) is used, and the fitness function is defined as  
\[\text{Fitness}=\frac{1}{\text{LMSE}}, \tag{59}\]  
and the LMSE is given as  
\[\text{LMSE}=\left(\sum_{k}(|H_{i}(e^{-j\omega_{k}})|-|H_{d}(e^{-j\omega_{k}})| )^{2}\right)^{\frac{1}{2}}, \tag{60}\]  
where \(H_{i}(e^{-j\omega_{k}})\) is the frequency response of ideal filter, and \(H_{d}(e^{-j\omega_{k}})\) is the frequency response of designed filter. Figure 8 shows the frequency responses of the FIR filters designed by GA and DE algorithms. The length of the two filters is \(N=21\), and Table 4 shows the control parameters of GA and DE algorithms, which have a significant effect on the performance of these two algorithms. As can be seen from Figure 8, the FIR filters designed by GA and DE algorithms have similar performance, that is, the stopband attenuation using GA is slightly better than that using DE. But the convergence speed of the DE algorithm is faster than GA.  
```
0: Initialize parameters: population size \(NP\), current generation \(g=1\), maximum generation \(G_{max}\), dimension \(D\), tolerance \(\varepsilon\), swarm \(S\), base vector \(s\), mutant vector \(v\), trial vector \(u\) ;
0: The best resolution \(BS\); for\(i=1\) to \(NP\)do for\(d=1\) to \(D\)do \(s_{i,g}^{d}=s_{min}^{d}+rand\cdot(s_{max}^{d}-s_{min}^{d})\); endfor endfor while\((|f(BS)|\geq\varepsilon)\) or \((g\leq G_{max})\)do for i=1 to NPdo ford=1 to D do \(v_{i,g}^{d}=\text{Mutation}(s_{i,g}^{d})\); \(u_{i,g}^{d}=\text{Crossover}(s_{i,g}^{d},v_{i,g}^{d})\); endfor if\(f(u_{i,G})<f(s_{i,g})\)then \(s_{i,g}=u_{i,g}\); if\(f(s_{i,g})<f(BS)\)then \(BS=s_{i,g}\); endif else \(s_{i,



#### 3.3.6. [PERSON]'s Method

In Reference [98], the prototype filter is designed by directly optimizing the filter coefficients. This method is mainly designed for FBMC based systems, aiming to minimize the stopband energy and constrain the ISI/ICI. The optimization problem is formulated as

\[\min_{h(0),h(1),\cdots h(N-1)}\int_{\omega_{0}}^{T}|H(e^{i\omega})|^ {2}d\omega\] (61) s.t. \[\left\{ \begin{array}{ll}h(n)=h(N-n-1),&\text{C1}\\ \text{ISI and ICI}\leq T_{1},&\text{C2}\\ \sum_{n=0}^{N-1}(h(n))^{2}=1,&\text{C3}\end{array} \right.\]

where \(T_{1}\) is a threshold. However, this optimization problem is non-convex and non-linear, which greatly increases the computational complexity. Through a series of approximation of the constraints, the number of unknowns in the optimization problem is enormously reduced. Therefore, the optimization problem can be solved and the computational complexity becomes acceptable. #### 3.3.7. Hunziker's Method

In Reference [136], the design of filter banks for maximal time-frequency (TF) resolution is proposed. Two properties should be satisfied to compute the optimal DFT filter banks. Firstly, the window under the filter operation is orthogonal, thus for white input processes, the sampling at the output of the filter banks generates uncorrelated random variables. Secondly, to ensure the minimum leakage of signal components outside the target area in TF plane, the prototype window shows the best TF localization feature. To satisfy the first property, the parametrization of paraunitary filter banks in Reference [137] is used and longer pulses are extended in Reference [138]. As for the second property, the Rihaczek distribution is used in TF region, and the objective function along with the constraints is transformed into a form which can be solved by semi-definite programming (SDP) algorithm. #### 3.3.8. [PERSON]'s Method

In Reference [139], the FIR filter design based on SDP is proposed.