## 3 Method
### Reinforcement Learning to Optimize Fusion Parameters
#### 3.3.3 Q-Learning in RL-Fusion  
Q-Learning computes the reward value by comparing the new result \(\mathrm{E}(\mathrm{P})\), from the SOTA LLM evaluation (calculated using Equation (5)), with the previous result stored in the Q-table (calculated using Equation (10)). Based on this reward, the algorithm proceeds to action selection and subsequently updates the Q-value accordingly. The main steps of Q-learning in RL-Fusion are shown in Figure 3.  
Figure 3: Main steps of Q-learning in RL–Fusion. The reward function offers feedback to the agent, directing its decision-making process. A positive reward is assigned when a new model configuration improves performance (accuracy increase), with the reward magnitude proportional to the improvement (+10 for a 10% increase in accuracy). Conversely, a negative reward is applied when performance declines (accuracy decrease), penalizing the agent for suboptimal choices (\(-10\) for a 10% reduction in accuracy). The reward function can be formally described as follows:  
\[r_{t}=\theta\cdot\Delta P\cdot 1_{\Delta P}, \tag{10}\]  
where \(r_{t}\) is the reward received after the \(t\)-th decision; \(\theta\) is a scaling factor for the reward, which can be set to 10 to reflect a 10% accuracy gain; \(\Delta P\) is the performance change \(\mathrm{E}(\mathrm{P})\) new and \(\mathrm{E}(\mathrm{P})\) old (calculated using Equation (5)); \(1_{\Delta P}\) is the indicator function when performance improves which is 1 if \(\Delta P\) > 0 and \(-1\) otherwise. The reward function directs the agent's model configuration decisions by assigning positive or negative rewards based on performance improvement or degradation. The reward magnitude correlates with the change in performance, and a model comparison-based mechanism ensures that the agent is optimizing in the correct direction. active selection  
By selecting different actions (Table 1), the agent progressively approaches the optimal fusion configuration by choosing under-explored actions in conjunction with the exploration factor, based on the updated Q-value optimization parameters. The specific action selection process is described in Equation (11):  
\[\mathrm{action}=\left\{ \begin{aligned} \mathrm{random\ choice\ with\ probability}\ \varepsilon\\ \mathrm{argmax}_{\mathrm{a}}\mathrm{Q}(\mathrm{state},\mathrm{a})\ \mathrm{with\ probability}\ 1-\varepsilon\end{aligned} .\right. \tag{11}\]  
Through \(\varepsilon-\)greedy policy, the agent always selects the action with the highest current Q value, aiming to maximize the expected reward. With probability \(\varepsilon\), a random action is selected, and with probability \(1-\varepsilon\) (Determination based on \(r_{t}\) Equation (10)), the action with the highest Q value is chosen. This method balances exploration and exploitation, allowing the agent to try different actions during the learning process and avoid getting stuck in local optima. The algorithm will, with a certain probability, select parameter configurations that have performed well (greedy selection), and with a certain probability, randomly explore new parameter configurations to prevent getting stuck in local optima.