## 2 System Model and Problem Formulation
Similarly, the corresponding digital baseband combiner and precoder are denoted by \(\mathbf{G}_{BB1,k}\in\mathbb{C}^{N_{rd}^{RF}\times N_{s}}\) and \(\mathbf{G}_{BB2,k}\in\mathbb{C}^{N_{rd}^{RF}\times N_{s}}\), respectively. These analog and digital processing units (\(\mathbf{F}_{1k}\), \(\mathbf{F}_{2k}\), \(\mathbf{G}_{BB1,k}\), \(\mathbf{G}_{BB1,k}\)) constitute the hybrid filter \(\mathbf{F}_{k}\in\mathbb{C}^{N_{rd}\times N_{rd}}\) at the \(k\)th relay node. The mathematical formulation of \(\mathbf{F}_{k}\in\mathbb{C}^{N_{rd}\times N_{rd}}\) is written as  
\[\mathbf{F}_{k}=\mathbf{F}_{2k}\mathbf{G}_{BB2,k}\mathbf{G}_{BB1,k}^{H}\mathbf{ F}_{1k}^{H}=\mathbf{F}_{2k}\mathbf{G}_{BB,k}\mathbf{F}_{1k}^{H}, \tag{5}\]  
where \(\mathbf{G}_{BB,k}=\mathbf{G}_{BB2,k}\mathbf{G}_{BB1,k}^{H}\in\mathbb{C}^{N_{rd }^{RF}\times N_{rd}^{RF}}\) is the overall digital baseband processing matrix at the \(k\)th relay node. When receive processing is applied on (4) then, the processed signal can be characterized as  
\[\mathbf{y}_{1,k}=\mathbf{G}_{BB1,k}^{H}\mathbf{F}_{1k}^{H}\mathbf{H}_{k}\ \mathbf{V}_{RF}\mathbf{V}_{BB,k}\mathbf{s}_{k}+\mathbf{G}_{BB1,k}^{H}\mathbf{F }_{1k}^{H}z_{r}\in\mathbb{C}^{N_{s}\times 1}. \tag{6}\]  
While designing a hybrid precoder at the source and a hybrid combiner at the relay node, it is desired to minimize interference among transmitted data streams from the source to relay node, which in turn maximizes SNR at the relay node. This task can be achieved by transforming the composite channel \(\mathbf{H}_{comp1}=\mathbf{G}_{BB1,k}^{H}\mathbf{F}_{1k}^{H}\mathbf{H}_{k}\ \mathbf{V}_{RF}\mathbf{V}_{BB,k}\in\mathbb{C}^{N_{s}\times N_{s}}\) into an equivalent parallel single-input single-output (SISO) channels. Using this formation, the received signal at relay corresponding to the \(n\)th data stream \(s_{n,k}\) in \(\mathbf{s}_{k}\) (1) can be represented with the following input output relationship  
\[y_{1,k}^{n}=\alpha_{n,k}\ s_{n,k}+v_{1,k},\ n\in\{1,2,\ldots,N_{s}\}, \tag{7}\]  
where \(\alpha_{n,k}\triangleq\|\mathbf{G}_{BB1,k}^{H}(\cdot,n)\mathbf{F}_{1k}^{H} \mathbf{H}_{k}\ \mathbf{V}_{RF}\mathbf{V}_{BB,k}(\cdot,n)\|\) and the operator \(\|.\|\) represent the absolute value of a complex number and, \(v_{1,k}\sim\mathcal{CN}(0,\ \sigma_{r}^{2})\). Therefore, (7) can be expressed as  
\[y_{1,k}^{n}=\|\mathbf{G}_{BB1,k}^{H}(\cdot,n)\mathbf{F}_{1k}^{H}\mathbf{H}_{k} \ \mathbf{V}_{RF}\mathbf{V}_{BB,k}(\cdot,n)\|\ s_{n,k}+v_{1,k}. \tag{8}\]  
Relay node performs hybrid beamforming on \(\mathbf{y}_{1,k}\) (6) for further transmission of this signal to the destination. The received signal at the destination can be written as  
\[\mathbf{y}_{2,k}^{\sim}=\mathbf{G}_{k}\mathbf{F}_{2k}\mathbf{G}_{BB2,k} \mathbf{y}_{1,k}+z_{d}\in\mathbb{C}^{N_{r}\times 1}. \tag{9}\]  
Finally, the received signal after applying hybrid combiner at the destination can be modeled as  
\[N_{s}\leq min\Big{(}N_{t}^{RF},\ N_{rel}^{RF},\ N_{r}^{RF}\Big{)}\ll min(N_{t}, \ N_{rel},\ N_{r}). \tag{2}\]0  
where \(\mathbf{W}_{RF}\in\mathbb{C}^{N_{r}\times N_{r}^{RF}}\), \(\mathbf{W}_{BB,k}\in\mathbb{C}^{N_{r}^{RF}\times N_{s}}\), \(\mathbf{G}_{k}\in\mathbb{C}^{N_{r}\times N_{rd}}\), and \(z_{d}\in\mathbb{C}^{N_{r}\times 1}\) are the common analog RF combiner, digital baseband combiner corresponding to the \(k\)th selected relay, channel matrix between the \(k\)th relay node and the final destination and ZMCSCG noise vector with variance \(\sigma_{d}^{2}\) i.e., \(z_{d}\sim\mathcal{CN}(\mathbf{0},\ \sigma_{d}^{2}\mathbf{I}_{N_{r}})\), respectively. Similarly, it is also required to minimize interference, while designing a hybrid beamforming matrix at the relay node and destination, for efficient transmission of data streams from the selected relay node to the end user.