## 3 RSMA-Assisted Integrated Sensing and Communication Systems
### RSMA-Assisted ISAC Architecture  
In this section, we provide an overview of the interaction between RSMA and ISAC, as shown in Figure 4. Note that RSMA is applied in ISAC systems to control inter-function interference (i.e., interference between radar and communication functions) as well as intra-function interference (i.e., interference among communication users). Note that the key difference between RSMA-ISAC and RSMA is that the common stream of RSMA-ISAC serves to form a beam towards the radar target. Thus, the multi-functionality of the common stream in RSMA-ISAC enables efficient operation of joint sensing and communication even without a dedicated radar signal. In other words, RSMA-ISAC employs a dynamic interference management strategy to optimize the balance between sensing accuracy and communication reliability, making it more adaptable and robust to operational scenarios compared to RSMA. Recall that ISAC is expected to play a pivotal role in next-generation wireless networks, achieving both hyper-connectivity and reliable sensing capabilities. Note that ISAC incorporates wireless communication and remote sensing functionalities into a unified system, where both functions share the spectrum and hardware. Note that in the monostatic ISAC systems, BS has \(N_{t}\) transmit antennas and \(N_{r}\) receive antennas, enabling serving \(K\) users and detecting moving targets simultaneously. For simplicity, we consider a one-layer RSMA model as described in Section 3.1. Let \(\mathbf{s}[t]=[s_{1}[t],s_{2}[t],\ldots,s_{K}[t],s_{c}[t],s_{R}[t]]^{T}\in \mathbb{C}^{(K+2)\times 1}\) denote the baseband signal before the precoding is applied where \(t\in\mathcal{T}=\{1,2,\ldots,T\}\) denotes the discrete-time index within one coherent processing interval (CPI) and \(s_{R}[t]\) is a radar sequence. Note that both the transmitter and receiver typically know \(s_{R}\) in advance. If \(T\) is sufficiently large and the data  
Figure 3: RSMA tranceiver architecture. streams are independent, then \(\frac{1}{T}\sum_{t=1}^{T}\mathbf{s}[t]\mathbf{s}[t]^{H}=\mathbf{I}_{K+2}\) is satisfied. The transmit signal at time index \(t\) is expressed as  
\[\mathbf{x}[t]=\mathbf{Ps}[t]=\mathbf{p}_{R}s_{R}[t]+\mathbf{p}_{c}s_{c}[t]+ \sum_{k\in\mathcal{K}}\mathbf{p}_{k}s_{k}[t] \tag{5}\]  
where the precoding matrix \(\mathbf{P}=[\mathbf{p}_{1},\mathbf{p}_{2},\ldots,\mathbf{p}_{K},\mathbf{p}_{ c},\mathbf{p}_{R}]\in\mathbb{C}^{N_{t}\times(K+2)}\) is maintained over one CPI. To be specific, \(\{\mathbf{p}_{1},\mathbf{p}_{2},\ldots,\mathbf{p}_{K}\}\) are the precoding vectors for \(K\) private streams while \(\mathbf{p}_{c}\) and \(\mathbf{p}_{R}\) are the precoding vectors for the common stream and radar sequence, respectively. The covariance matrix of the transmit signal can be written as \(\mathbf{R}_{x}=\frac{1}{T}\sum\limits_{t=1}^{T}\mathbf{x}[t]\mathbf{x}[t]^{H} =\mathbf{P}\mathbf{P}^{H}\). Note that the transmit power budget is \(P_{t}\) (i.e., \(\text{tr}\big{(}\mathbf{P}\mathbf{P}^{H}\big{)}\leq P_{t}\)). For the received signal in the communication model, each user receives the signal as follows  
\[ \begin{split} y_{k}[t]&=\mathbf{h}_{k}^{H}\mathbf{ x}[t]+n_{k}[t]\\ &=\mathbf{h}_{k}^{H}\mathbf{p}_{R}s_{R}[t]+\mathbf{h}_{k}^{H} \mathbf{p}_{c}s_{c}[t]+\mathbf{h}_{k}^{H}\sum_{k\in\mathcal{K}}\mathbf{p}_{k} s_{k}[t]+n_{k}[t].\end{split}  \tag{6}\]  
Firstly, each user decodes the common stream by treating other streams as interference. The achievable common rate for the \(k\)-th user can be expressed as  
\[R_{c,k}=\text{log}_{2}\Bigg{(}1+\frac{|\mathbf{h}_{k}^{H}\mathbf{p}_{c}|^{2}} {\sum_{i\in\mathcal{K}}|\mathbf{h}_{k}^{H}\mathbf{p}_{i}|^{2}+\delta|\mathbf{ h}_{k}^{H}\mathbf{p}_{R}|^{2}+\sigma_{k}^{2}}\Bigg{)} \tag{7}\]  
where \(\delta\) is the binary parameter related to the SIC operation of the radar sequence. When the SIC for \(s_{R}\) is applied, \(\delta\) becomes 0. After the common stream is subtracted from the received signal, each user decodes the private stream.