## 4 Problem Formulation and Algorithm Design
### Digital Beamforming Design  
In this section, we optimize the digital beamforming vectors \(\mathbf{v}_{1},\mathbf{v}_{2}\) given the holographic beamforming vector \(\mathbf{m}\). Let \(\tilde{\mathbf{h}}_{u}^{H}=\mathbf{m}^{H}\hat{\mathbf{H}}_{u}\) and \(\tilde{\mathbf{h}}_{\epsilon_{k}}^{H}=\mathbf{m}^{H}\hat{\mathbf{H}}_{\epsilon_ {k}}\), and then we denote \(\tilde{\mathbf{H}}_{u}=\tilde{\mathbf{h}}_{u}\tilde{\mathbf{h}}_{u}^{H}\) and \(\tilde{\mathbf{H}}_{\epsilon_{k}}=\tilde{\mathbf{h}}_{\epsilon_{k}}\tilde{ \mathbf{h}}_{\epsilon_{k}}^{H}\). Therefore, (P0) can be transformed to  
\[ \begin{array}{ll}\text{(P1.1)}:\max_{\mathbf{v}_{1},\mathbf{v}_{2}}&\left\{ \log_{2}\!\left(1+\frac{\mathbf{v}_{1}^{H}\tilde{\mathbf{H}}_{u}\mathbf{v}_{ 1}}{J\sigma^{2}+\mathbf{v}_{2}^{H}\tilde{\mathbf{H}}_{u}\mathbf{v}_{2}} \right)-\max_{k}\log_{2}\!\left(1+\frac{\mathbf{v}_{1}^{H}\tilde{\mathbf{H}}_ {\epsilon_{k}}\mathbf{v}_{1}}{J\sigma^{2}+\mathbf{v}_{2}^{H}\tilde{\mathbf{H} }_{\epsilon_{k}}\mathbf{v}_{2}}\right)\right\}\\ &\text{s.t.}&\mathbf{v}_{1}^{H}\mathbf{v}_{1}+\mathbf{v}_{2}^{H}\mathbf{v}_{ 2}\leq P_{\text{max}}\,.\end{array}  \tag{10}\]  
Note that \(\mathbf{v}_{1}^{H}\tilde{\mathbf{H}}_{l}\mathbf{v}_{1}=\mathbf{Tr}\big{(} \tilde{\mathbf{H}}_{l}\mathbf{v}_{1}\mathbf{v}_{1}^{H}\big{)}\) and \(\mathbf{v}_{2}^{H}\tilde{\mathbf{H}}_{l}\mathbf{v}_{2}=\mathbf{Tr}\big{(} \tilde{\mathbf{H}}_{l}\mathbf{v}_{2}\mathbf{v}_{2}^{H}\big{)},l\in\{u,e_{k}\}\). We define two matrices \(\mathbf{V}_{1}=\mathbf{v}_{1}\mathbf{v}_{1}^{H}\) and \(\mathbf{V}_{2}=\mathbf{v}_{2}\mathbf{v}_{2}^{H}\), and then it follows that \(\mathbf{V}_{1}\succeq 0,\mathbf{V}_{2}\succeq 0\) and \(rank(\mathbf{V}_{1})=rank(\mathbf{V}_{2})=1\). Since the rank-1 constraint is a non-convex constraint, we use the semidefinite relaxation (SDR) method to remove this constraint. Then, the (P1.1) can be transformed to  
\[ \begin{array}{ll}\text{(P1.2):}\max_{\mathbf{V}_{1},\mathbf{V}_{2}}&\left\{ \log_{2}\!\left(1+\frac{\mathbf{Tr}\big{(}\tilde{\mathbf{H}}_{u}\mathbf{V}_{ 1}\big{)}}{J\sigma^{2}+\mathbf{Tr}\big{(}\tilde{\mathbf{H}}_{u}\mathbf{V}_{2} \big{)}}\right)-\max_{k}\log_{2}\!\left(1+\frac{\mathbf{Tr}\big{(}\tilde{ \mathbf{H}}_{\epsilon_{k}}\mathbf{V}_{1}\big{)}}{J\sigma^{2}+\mathbf{Tr}\big{(} \tilde{\mathbf{H}}_{\epsilon_{k}}\mathbf{V}_{2}\big{)}}\right)\right\}\\ &\text{s.t.}&(\mathbf{V}_{1},\mathbf{V}_{2})\in\mathcal{V},\end{array}  \tag{11}\]  
where \(\mathcal{V}=\{(\mathbf{V}_{1},\mathbf{V}_{2})|\mathbf{Tr}(\mathbf{V}_{1}+ \mathbf{V}_{2})\leq P_{max},\mathbf{V}_{1}\succeq 0,\mathbf{V}_{2}\succeq 0\}\) is the feasible set for \((\mathbf{V}_{1},\mathbf{V}_{2})\). This problem is still difficult to solve because the objective function is not jointly concave with respect to \((\mathbf{V}_{1},\mathbf{V}_{2})\). In order to overcome this problem, we propose the following Lemma 1[23]. **Lemma 1**. : _Consider the function \(\varphi(t)=-tx+\ln(t)+1\), for any \(x>0\), then we have_  
\[-\ln(x)=\max_{t>0}\varphi(t); \tag{12}\]  
_the optimal solution is \(t=1/x\)._  
The upper bound for \(\varphi(t)\) can be given by Lemma 1. By setting \(x=\mathbf{Tr}\big{(}\tilde{\mathbf{H}}_{u}\mathbf{V}_{2}\big{)}+J\sigma^{2}\) and \(t=t_{u}\) as well as applying Lemma 1, we have  
\[ \begin{array}{ll}R_{u}\ln 2&=\ln\!\left(\mathbf{Tr}\big{(}\tilde{\mathbf{H}}_{u}( \mathbf{V}_{1}+\mathbf{V}_{2})\big{)}+J\sigma^{2}\right)-t_{u}\Big{(}\mathbf{ Tr}\big{(}\tilde{\mathbf{H}}_{u}\mathbf{V}_{2}\big{)}+J\sigma^{2}\Big{)}\\ &=\max_{t_{u}>0}\varphi_{u}(\mathbf{V}_{1},\mathbf{V}_{2},t_{u}),\end{array}  \tag{13}\]  
where \(\varphi_{u}(\mathbf{V}_{1},\mathbf{V}_{2},t_{u})=\ln\!\left(\mathbf{Tr}\big{(} \tilde{\mathbf{H}}_{u}(\mathbf{V}_{1}+\mathbf{V}_{2})\big{)}+J\sigma^{2} \right)-t_{u}\big{(}\mathbf{Tr}\big{(}\tilde{\mathbf{H}}_{u}\mathbf{V}_{2} \big{)}+J\sigma^{2}\big{)}+\ln(t_{u})+1\). This function is jointly concave with respect to \((\mathbf{V}_{1},\mathbf{V}_{2})\).