## 4 DRL-Based Joint Task Offloading and Resource Allocation Scheme
### MDP Formulation  
We use a four-elements tuple \((\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})\) to denote the MDP, where \(\mathcal{S}\) and \(\mathcal{A}\) denote the set of system state and actions, respectively. \(\mathcal{P}\) is the state transition probability and \(\mathcal{R}\) represents the reward function. We can outline the process of RL interacting with the environment as follows. The agent adopts action \(a_{t}\) under environment state \(s_{t}\), and receives the instant reward \(r_{t}\) as the response for the action \(a_{t}\). Then, the environment state \(s_{t}\) turns to new \(s_{t}\) according to the transition function \(\mathcal{P}(s_{t},a_{t},s_{t+1})\). The reinforcement learning aims to obtain the optimal policy \(\pi^{*}(a\mid s)\) from a given MDP, which is the mapping from state to action that can obtain the maximum long-term cumulative reward \(R_{t}=\sum_{i=0}^{\infty}\gamma^{i}\mathcal{R}(s_{i+i+1},a_{t+i+1})\). \(\gamma\in[0,1)\) is the discount factor. We can define state, action, and reward in our model as follows.  
State: The environmental state at the \(t\)-th time step consists of channel matrices, BS transmit power, the size of the computation task, and the action adopted by the agent in \((t-1)\)-th time step. Thus, the state of agent \(s_{t}\in\mathcal{S}\) is given by  
\[s_{t}=\{\mathbf{\tilde{H}}_{1}(t),\mathbf{\tilde{H}}_{2}(t),\mathbf{p}(t), \mathbf{d}(t),\mathbf{a}(t-1)\}, \tag{30}\]  
where  
* \(\mathbf{\tilde{H}}_{1}(t)=[\mathrm{Re}\{\mathbf{H}_{1}(t)\},\mathrm{Im}\{ \mathbf{H}_{1}(t)\}]\): the channel matrix \(\mathbf{H}_{1}(t)\) is divided into the real part and imaginary part, due to the fact that the neural network cannot deal with the complex value.
* \(\mathbf{\tilde{H}}_{2}(t)=[\mathrm{Re}\{\mathbf{H}_{2}(t)\},\mathrm{Im}\{ \mathbf{H}_{2}(t)\}]\): as the same way, \(\mathbf{H}_{2}(t)\) is separated into two independent parts, and \(\mathbf{H}_{2}(t)=\{\mathbf{h}_{k,2}(t)|k\in\mathcal{K}\}\).
* \(\mathbf{p}(t)=\{[\mathrm{Re}\{p_{k}(t)\},\mathrm{Im}\{p_{k}(t)\}]\mid\forall k \in\mathcal{K}\}\): the transmit power for each UE and divided into two ports inputting the training network with \(p_{k}(t)=\mathrm{Tr}\big{(}\mathbf{w}_{k}\mathbf{w}_{k}^{H}\big{)}\).
* \(\mathbf{d}(t)=[d_{k}(t)\mid\forall k\in\mathcal{K}]\): the size of the computation task generated at UE.
* \(\mathbf{a}(t-1)\): denotes the action selected by the agent at the previous time step.  
Action: The action of the agent comprises the transmit beamforming matrix at BS, phase shift of IRS, and computation offloading decision. We can formulated the action \(a_{t}\in\mathcal{A}\) as  
\[a_{t}=\{\mathbf{W}(t),\mathbf{\Phi}(t),w_{k}(t)\}, \tag{31}\]  
where \(\mathbf{\hat{W}}(t)=[\mathrm{Re}\{\mathbf{W}(t)\},\mathrm{Im}\{\mathbf{W}(t)\}]\) and \(\mathbf{\hat{\Phi}}(t)=[\mathrm{Re}\{\mathbf{\Phi}(t)\},\mathrm{Im}\{\mathbf{ \Phi}(t)\}]\) indicate the real and imaginary parts of transmit beamforming and phase shift matrices. \(w_{k}\in[0,1]\) for \(\forall k\in\mathcal{K}\) is the computation offloading action.  
Reward: The agent, through the feedback of reward, evaluates the action and makes improvements. This work aims to optimize the communication data rate while minimizing the system energy consumption. Thus, the reward \(r_{t}\) at the \(t\)-th time step is defined by  
\[r_{t}=\omega_{1}\Psi_{1}(t)-\omega_{2}\Psi_{2}(t), \tag{32}\]  
where \(\omega_{1}\) and \(\omega_{2}\) are the weighting factors with \(\omega_{1}+\omega_{2}=1\). The weighting factor can be used for control optimization preferences.