## 5 Improved SAC (iSAC) Algorithm  
### Algorithm Architecture  
In DRL, an agent must learn the optimal policy through interactions with the environment. During this process, the agent faces two main tasks: exploration and exploitation. Exploration refers to the agent's attempt to try different actions to discover new, potentially better strategies, whereas exploitation refers to the agent's selection of actions based on the currently known optimal policy. The SAC algorithm balances these two tasks by maximizing an entropy. Entropy, a measure of the uncertainty or randomness of a system, is also used in information theory to gauge the uncertainty of information. In reinforcement learning algorithms that maximize entropy, the goal is not only to maximize accumulated rewards but also to maximize the entropy of the policy. This means that the algorithm is encouraged to explore a variety of different actions, even if these actions do not appear to be the optimal choice at the current moment. The original SAC algorithm achieves the maximization of the entropy by introducing an entropy term in the objective function. Specifically, the objective function of SAC consists of two parts: one is the expectation of accumulated rewards and the other is the expectation of the policy entropy. By adjusting the weights of these two parts, SAC can maximize accumulated rewards while maintaining the randomness of the policy, thus avoiding premature convergence to local optima. Equation (18) represents the overall maximization goal of SAC, where \(\gamma\) is the discount factor and \(\alpha\) is the hyperparameter that balances the trade-off between the rewards and the entropy, also known as the temperature coefficient. \[\max_{\pi}\mathbb{E}_{\tau\sim\gamma\pi}\Bigg{[}\sum_{t=0}^{\infty}\gamma^{t} r(s_{t},a_{t})\Bigg{]}+\alpha\mathbb{E}_{(s,\alpha)\sim\pi}[\log\pi(a|s)] \tag{18}\]  
In the SAC algorithm, the policy network is responsible for generating the probability distribution of actions, while the value network is responsible for estimating the state-action value function. The agent interacts with the environment according to the current policy, collecting information such as states, actions, rewards, and new states, which are used to update the value network and the policy network. When updating the policy network, SAC considers not only maximizing the expected reward but also maximizing the policy entropy, as shown in Figure 3. This is achieved by computing the policy gradient and updating the weights of the policy network. An important feature of the SAC algorithm lies in its ability to adaptively adjust the entropy weight. During the training process, entropy weight \(\alpha\) is automatically adjusted by another parameter, which allows SAC to automatically balance exploration and exploitation in different tasks and environments without manual interventions. Additionally, SAC is off-policy, which means it can utilize past data through a replay buffer for repeated training. This off-policy updating method offers greater stability compared to online algorithms like A3C. The improvements made in this study to the SAC algorithm mainly involve the following aspects:  
(1) Since the SAC algorithm is a reinforcement learning algorithm suitable for continuous action spaces and the task offloading problem in IoV belongs to discrete space decision making, this paper proposes iSAC, which is applicable to discrete space decisions. As shown in Figure 3, the original actor network only had fully connected layers and ReLU activation functions. This paper adds normalization layers and softmax activation functions. The normalization layer standardizes the output, thereby enhancing the model's representing power. The main reason for using a normalization layer in the output is to prevent thesoftmax function's output from being singular. The softmax activation function, as defined in Equation (19), transforms the policy output by SAC into a probability distribution vector, then edge servers can be sampled randomly according to these probabilities.