# A Multi-Task Dynamic Weight Optimization Framework Based on Deep Reinforcement Learning
## 3 Methodology
### Deep Reinforcement Learning Agents Based on Experience Replay: An Actor-Critic Approach  
To achieve dynamic optimization of task weights in a multitask learning environment with high-dimensional and non-linear features, we introduce a deep reinforcement learning agent based on experience playback. Inspired by the shared-memory-based multi-task deep learning approach in the literature [38], this agent combines the advantages of deep learning and reinforcement learning, adopts the Actor-Critic framework, and utilizes the experience playback mechanism to improve the training stability and sample utilization efficiency. The DRL agent outputs new task weights based on the current state information by interacting with the multi-task models \(\{\alpha_{t}\}\). In the implementation, the experience playback buffer is used to store the transferred samples during the interaction \((s,\alpha,r,s^{\prime},d)\), where \(s\) is the current state, \(\alpha\) is the current action (i.e., newly generated task weights), \(r\) is the reward, \(s^{\prime}\) is the next state, and \(d\) is the termination marker. By randomly sampling historical experiences from the buffer, the DRL agent can make full use of past information and reduce the frequency of direct interaction with the environment, thus improving training efficiency and policy stability. **1. State, Action, and Reward Design**  
**State Representation**:  
\[s=[\mathbf{A},\sigma_{A},\alpha_{\text{prev}}] \tag{2}\]  
where \(\mathbf{A}=[A_{1},A_{2},\dots,A_{T}]\) denotes the accuracy rate of each task, \(\sigma_{A}\) is the standard deviation of the accuracy rates to reflect the imbalance of the task performance, and \(\alpha_{\text{prev}}\) is the weight of the task in the previous round. This state representation provides the agent with comprehensive task performance information and reference. **Action Definition**: Action \(\mathbf{\alpha}\) is the new task weight output by the agent, which needs to satisfy \(\alpha_{t}\geq\alpha_{\text{min}}\) and \(\sum_{t=1}^{T}\alpha_{t}=1\). By dynamically adjusting the task weights, the agent can influence the learning process of the multi-task model, thus achieving adaptive control of the inter-task balance. **Reward Function**:  
\[\text{Reward}=\left(\prod_{i=1}^{N}A_{i}\right)^{\frac{1}{N}}+\lambda\cdot \left(\frac{1}{N}\sum_{i=1}^{N}A_{i}\right) \tag{3}\]  
Figure 3: Schematic structure of multi-task classification network for dual-task classification scenario. where \(\left(\prod_{i=1}^{N}A_{i}\right)^{\frac{1}{N}}\) is the geometric mean of the task accuracy, which is more sensitive to low-accuracy tasks and thus incentivizes the boosting of the poorest-performing tasks, and \(\lambda\) is a trade-off parameter that balances the importance of geometric versus arithmetic averages in the reward. This reward design not only encourages overall performance improvement but also ensures that attention is paid to the weaker tasks, thus preventing imbalance between tasks. **2. Actor-Critic Algorithm Based on Experience Playback**  
When training DRL agents, we use the Actor-Critic algorithm based on experience playback. The specific steps are as follows:  
**Critic Network Update**:  
First, calculate the target value:  
\[y=r+\gamma\cdot Q(s^{\prime},\alpha^{\prime}) \tag{4}\]  
where \(r\) is the reward, \(\gamma\) is the discount factor, \(s^{\prime}\) is the next state, and \(\alpha^{\prime}\) is the next action. Estimate \(Q(s,\alpha)\) by minimizing the Mean Square Error (MSE) loss between the current value of the Critic network \(Q(s,\alpha)\) and the target value \(y\):  
\[L_{\text{critic}}=\text{MSE}(Q(s,\alpha),y) \tag{5}\]  
Continuously and iteratively update the parameters of the Critic network so that it more accurately evaluates the value of state-action pairs.