## 5 IICA-QL for EEMsMIAJSP
### Assimilation Based on Q-Learning Adaptive Adjustment of Operator Parameter
#### 5.2.1 Adaptive Adjustment of UX Operator Parameter Based on Q-Learning  
During the assimilation process, the uniform crossover (UX) operator [17] and the two-point Crossover (TPX) operator [31] are used, respectively applied to the movement of colonies towards the imperialist and the external archive set. Among these, the UX operator is used as the main evolutionary operator to guide the evolutionary process of population and can also be applied for local search [17], playing an important role in the quality of population evolution. The parameter of the UX operator determines the number of high-quality genes passed from parents to offspring. In the early stages of evolution, a larger parameter is helpful for the algorithm convergence, while in the later stages of evolution, a larger parameter can result in poor population diversity, causing the algorithm to fall into local optima. Therefore, a Q-learning-based adaptive adjustment strategy for the UX operator parameter is devised to dynamically adjust the evolutionary process of the population, enabling the algorithm to converge more efficiently to the Pareto front. State: The state of the entire population is determined by evaluating the convergence metric \(\Delta GD\) of the non-dominated solution set compared to the reference set and the diversity metric \(\Delta Ratio\), which represents the coverage ratio of the solution space. Here, \(\Delta GD\) represents the generational distance difference in the non-dominated solutions obtained compared to the Pareto front \(PF^{*}\) between generation \(t\) and \(t+1\). \(PF^{*}\) is a preset reference point, and \(\Delta Ratio\) represents the proportion of subregions occupied by the non-dominated solutions in the current generation in the objective space. The objective space is divided into several non-overlapping regions by weight vectors [32]. \(\Delta GD\) indicates the degree of convergence of generation \(t+1\) compared to generation \(t\); the smaller the value of \(\Delta GD\), the better the convergence. \(\Delta Ratio\) represents the change in non-dominated set diversity, with \(\Delta Ratio<0\) indicating a decrease in diversity and \(\Delta Ratio>0\) indicating an increase in diversity. The calculations for \(\Delta GD\) and \(\Delta Ratio\) are given by Equations (24) and (25), respectively, where \(|PF_{t}|\) and \(|\psi_{t}|\) represent the number of non-dominated solutions in generation \(t\) and the proportion of the objective space they occupy. During the evolution, there are four possible states: (1) \(\Delta GD>=0\), \(\Delta Ratio>0\); (2) \(\Delta GD>=0\), \(\Delta Ratio<=0\); (3) \(\Delta GD<0\), \(\Delta Ratio>0\); and (4) \(\Delta GD<0\), \(\Delta Ratio<=0\). \[\Delta GD=(\sqrt{\sum_{x\in PF_{t+1}}\min_{y\in PF^{*}}{d(x,y)}^{2}}/{|PF_{t+1 }|})-(\sqrt{\sum_{x\in PF_{t}}\min_{y\in PF^{*}}{d(x,y)}^{2}}/{|PF_{t}|}) \tag{24}\]  
\[\Delta Ratio=|\psi_{t+1}|-|\psi_{t}| \tag{25}\]  
Action: The action set \(\mathbb{A}\) is composed of ten actions, representing ten levels of the UX operator parameter. By determining the current state of the population, the action with the highest Q-value in the Q-table is selected based on the \(\varepsilon\)-greedy strategy. \(\varepsilon\)-greedy: In the initial iterations, Q-values usually lack prior knowledge, making it easy for the agent to fall into local optima during learning. Therefore, an \(\varepsilon\)-greedy strategy based on a sigmoid function is used to guide the learning of the agent. The mathematical expression for \(\varepsilon\)-greedy is as follows:  
\[\varepsilon=0.1+(\varepsilon-0.1)\times\frac{1}{1+e^{(-0.2\times(g-50))}} \tag{26}\]  
where g represents the number of iterations. Reward: By calculating the values of \(\Delta GD\) and \(\Delta Ratio\) after executing an action, the agent receives a corresponding reward and updates the Q-value of the action in the Q-table for the corresponding state according to Equation (27).