## 4 Hybrid Action Space SAC-Based MLLM Task Offloading and Resource Allocation Scheme
### SAC Framework  
To optimize the decision variables in alignment with the MDP defined for Equation (34), we utilize the SAC algorithm to identify effective solutions. The SAC algorithm, an actor-critic method, integrates policy-based and value-based strategies, employing four neural networks: a value function network parameterized by \(\phi\), a soft Q-function network parameterized by \(\theta\), a policy network parameterized by \(\theta\), and a target value function network denoted by \(\phi\). Then, we clarify the roles of different Q-function networks:  
* \(Q_{\phi}(s(t))\): the state value function network that estimates the expected return from state \(s(t)\) under the entropy-regularized policy.
* \(Q_{\theta}(s(t),a(t))\): the soft Q-function network that evaluates state-action pairs, incorporating entropy bonuses in the value estimation.
* \(Q_{\phi}(s(t))\): the target value function network, updated slowly for training stability.
* \(\tilde{Q}(s(t),a(t))\): the target Q-value computed using the Bellman equation: \(\tilde{Q}(s(t),a(t))=r(s(t),a(t))+\gamma Q_{\tilde{\phi}}(s(t+1))\).  
In practice, these networks require training to implement the SAC algorithm. The update rule for the value function is given as  
\[J_{Q_{\phi}}=\mathbb{E}\bigg{[}\frac{1}{2}\big{(}Q_{\phi}(s(t))-\mathbb{E}[Q_ {\theta}(s(t),a(t))\ -\log\pi_{\theta}(a(t)|s(t))]\big{)}^{2}\bigg{]}\quad, \tag{35}\]  
where the Q-value associated with a given network is denoted by \(Q\)., the expectation operator by \(\mathbb{E}[\cdot]\), and the sample distribution by \(\mathcal{D}\). The SAC algorithm minimizes the loss by leveraging the squared residual error for the value function. The gradient of the value function loss, as derived from Equation (35), is expressed as  
\[\
abla_{\phi}J_{Q_{\phi}}= \
abla_{\phi}Q_{\phi}(s(t))(Q_{\phi}(s(t))-Q_{\theta}(s(t),a(t))+ \log\pi_{\theta}(a(t)|s(t))). \tag{36}\]  
Additionally, the loss function for updating the soft Q-network \(Q_{\theta}\) is formulated as  
\[J_{Q_{\theta}}=\mathbb{E}\bigg{[}\frac{1}{2}\big{(}Q_{\theta}(s(t),a(t))- \tilde{Q}(s(t),a(t))\big{)}^{2}\bigg{]}, \tag{37}\]  
where the target Q-value \(\tilde{Q}(s(t),a(t))\) is defined as  
\[\tilde{Q}(s(t),a(t))=r(s(t),a(t))+\gamma\mathbb{E}\Big{[}Q_{\tilde{\phi}}(s(t+ 1)),\Big{]}, \tag{38}\]  
with \(\gamma\) representing the discount factor and \(\rho\) denoting the state transition probability.  
The corresponding gradient for the Q-network update is given by  
\[\
abla_{\theta}J_{Q_{\theta}}=\
abla_{\theta}Q_{\theta}(s(t),a(t))(Q_{\theta} (s(t),a(t))-r(s(t),a(t))-\gamma Q_{\tilde{\phi}}(s(t+1))). \tag{39}\]Furthermore, the policy network is reformulated through a neural network transformation defined as \(a(t)=f_{\theta}(\xi(t);s(t))\), where \(\xi(t)\) represents a noise term adhering to a Gaussian distribution. The loss function for updating the policy network \(\pi_{\theta}\) is computed as  
\[J_{\pi_{\theta}}= \mathbb{E}\bigg{[}\log\pi_{\theta}(f_{\theta}(\xi(t);s(t))|s(t))- Q_{\theta}(s(t),f_{\theta}(\xi(t);s(t)))\bigg{]}, \tag{40}\]  
where \(\phi\) signifies the normal distribution. The corresponding gradient is calculated by  
\[\
abla_{\theta}J_{\pi_{\theta}}= \
abla_{\theta}\log\pi_{\theta}(a(t)|s(t))+\big{(}\
abla_{a(t)} \log\pi_{\theta}(a(t)|s(t))- \tag{41}\] \[\
abla_{a(t)}Q_{\theta}(s(t),a(t))\big{)}\
abla_{\theta}f_{ \theta}(\xi(t);s(t)),\]  
with the Q-value updated as  
\[Q(s(t),a(t))= r(s(t),a(t))+\gamma\mathbb{E}\Big{[}Q_{\hat{\phi}}(s(t+1)) \Big{]}, \tag{42}\]  
where \(\gamma\) denotes the discount factor, and \(\rho\) represents the state transition probability.