## 4 Hybrid Action Space SAC-Based MLLM Task Offloading and Resource Allocation Scheme
### Continuous-Discrete Action Space-Based SAC Algorithm  
In the optimization challenges delineated in problem \(\mathcal{P}_{1}\), the decision-making process involves a combination of discrete and continuous actions, presenting a substantial obstacle for traditional DRL algorithms [36]. Conventional techniques frequently address this issue by converting the continuous action space into a discrete form, which significantly expands the size of the action space. Alternatively, some methods attempt to reframe discrete actions as continuous, a strategy that increases complexity and obstructs the training process's ability to converge effectively. To mitigate these difficulties, we propose the integration of an action decoupling algorithm within the SAC framework through decomposing the Q-functions into discrete and continuous components:  
* _Discrete agent networks_: \(Q_{\mu}^{\theta}\), \(Q_{\mu}^{\theta}\), \(Q_{\mu}^{\theta}\) handle discrete actions (offloading decisions), where subscript \(\mu\) denotes discrete components. * _Continuous agent networks_: \(Q_{\
u}^{\theta}\), \(Q_{\
u}^{\theta}\) handle continuous actions (power allocation and UAV trajectories), where subscript \(\
u\) denotes continuous components. This approach entails a redefinition of both the action and the policy, enabling more efficient management of the hybrid action space. The policy function and action can be redefined as  
\[Q_{\theta}(a|s)=Q_{\mu}^{\theta}(a_{\mu}|s)\cdot Q_{\
u}^{\theta}(a_{\
u}|s)= \prod_{a_{i}\in a_{\mu}}Q_{\mu}^{\theta}(a_{i}|s)\cdot\prod_{a_{j}\in a_{\
u}} Q_{\
u}^{\theta}(a_{j}|s), \tag{43}\]  
\[A=\{\mu,\
u\}, \tag{44}\]  
where, \(\mu\) represents discrete actions, and \(\
u\) denotes continuous actions. Here, \(Q_{\theta}(a|s)\) represents the joint policy function obtained by combining the discrete and continuous policy components. This factorization enables independent learning of discrete and continuous action policies while maintaining their interdependence through the joint Q-value estimation. Accordingly, we develop two independent agents, each utilizing a distinct SAC framework to optimize the discrete or continuous variables of problem \(\mathcal{P}_{1}\). Consequently, the cost functions for the value networks of the discrete and continuous agents are expressed as  
\[J_{Q_{\mu}^{\theta}}= \mathbb{E}\bigg{[}\frac{1}{2}\big{(}Q_{\mu}^{\theta}(s(t))- \mathbb{E}\big{[}Q_{\mu}^{\theta}(s(t),a(t))-\log Q_{\mu}^{\theta}(a(t)|s(t)) \big{]}\big{)}^{2}\bigg{]}\quad, \tag{45}\]\[\hat{J}_{Q^{\theta}_{v}}= \mathbb{E}\bigg{[}\frac{1}{2}\big{(}Q^{\theta}_{v}(s(t))-\mathbb{E} \big{[}Q^{\theta}_{v}(s(t),a(t))-\log Q^{\theta}_{v}(a(t)|s(t))\big{]}\big{)}^{2} \bigg{]}\quad. \tag{46}\]  
Then, the cost functions for the soft Q-networks associated with discrete and continuous agents are defined as  
\[J_{Q^{\theta}_{v}}=\mathbb{E}\bigg{[}\frac{1}{2}\Big{(}Q^{\theta}_{\mu}(s(t),a (t))-\overline{Q}_{\mu}(s(t),a(t))\Big{)}^{2}\bigg{]}, \tag{47}\]  
\[J_{Q^{\theta}_{v}}=\mathbb{E}\bigg{[}\frac{1}{2}\Big{(}Q^{\theta}_{v}(s(t),a (t))-\overline{Q}_{v}(s(t),a(t))\Big{)}^{2}\bigg{]}. \tag{48}\]  
Additionally, the cost functions for the policy networks of the discrete and continuous agents are given by  
\[J_{Q^{\theta}_{\mu}}= \mathbb{E}\bigg{[}\log Q^{\theta}_{\mu}(f_{\theta}(\zeta(t);s(t) )|s(t))-Q^{\theta}_{\mu}(s(t),f_{\theta}(\zeta(t);s(t)))\bigg{]}\quad, \tag{49}\] \[J_{Q^{\theta}_{v}}= \mathbb{E}\bigg{[}\log Q^{\theta}_{v}(f_{\theta}(\zeta(t);s(t))|s (t))-Q^{\theta}_{v}(s(t),f_{\theta}(\zeta(t);s(t)))\bigg{]}\quad. \tag{50}\]  
To optimize the decoupled policies, we adopt the maximum a posteriori policy optimization algorithm, as described in [37]. For the hybrid action space within the SAC framework, we introduce a new policy \(\ell\) to maximize expected Q-values while satisfying Kullback-Leibler (KL) divergence constraints to ensure stable policy updates:  
\[J_{Q_{\ell}}=\mathbb{E}\big{[}\hat{Q}(s,a)\big{]}, \tag{51}\]  
where \(\hat{Q}\) represents the Q-function derived from the replay buffer \(\mathcal{C}\)[38].