## 3 Proposed Method
### Proposed Algorithm  
We use multiple actor-learners to exploit different explorations in parallel to maximize diversity rather than one actor-learner model. It reduces correlation in accordance with a time in a single learner's update method. Furthermore, it can shorten the training time and make the on-policy method more stable [32]. Algorithm 1 shows the asynchronous training process. Multiple subprocesses with their local model copy the global model's parameters to the local model before starting the following episode. After the end of each episode and learning process of Algorithm 2, the global model's parameters are updated to the trained local model's. ```
0:\(N\) (the number of items), \(M\) (the number of knapsacks), \(AL\) (the number of actor-learners)
0: Trained global model
1: Initialize a policy network \(\pi_{\theta}(a_{t}|s_{t})\) with parameters \(\theta\) for global model and value function \(V_{\theta_{v}}(s_{t})\) with parameters \(\theta_{v}\)
2:\(Process_{max}=AL,i=0\)
3:while\(i<Process_{max}\)do
4: Process \(i\) starts training process
5:endwhile
```  
**Algorithm 1** Asynchronous process  
Algorithm 2 runs on the process with the actor-critic model. Input is \(K\) problem instances with \(N\) items. The total iteration of training episodes is \(K\times R\) in line 1. In every iteration, episode \(i\) uses the \(e\)-th instance in the range between \(0\) and \((K-1)\) in line 3. The number of initial items at each episode is a random integer number between \(1\) and \(N\). The initial capacity of each knapsack is a random integer value between the minimum weight of the item sets and maximum capacity \(C\). While there is at least one candidate item, each step of an episode iterates. The episode is over when capacities are not sufficient, or overall items are chosen (selected or discarded), i.e., \(|I|=0\). An action is always chosen from the range between \(1\) and \(N\). In line 12, the agent does an action according to the policy \(\pi_{\theta}(a_{t}|s_{t})\). If the chosen item is not selected or discarded yet, and its weight is less than or equal to \(c_{1}\), then the item is selected for the knapsack. As a result, capacity \(c_{1}\) changes to current capacity minus the weight of the selected item according to action \(a\) (line 15). The item is classified as selected one and is excluded from the item set \(I\) (line 16 and 18). \(ai\) denotes an index of the initial item set. Subsequently, Algorithm 3 is called and constitutes the capacity features of the next state. The output of Algorithm 2 is decision variables of items for each instance in \(K\). Algorithm 3 is a greedy sort algorithm for the very next state following action in Algorithm 2. This sorts the capacities in the descending order, and it only deals with capacities that are larger than zero. Therefore, the process creates new sorted capacities \(c_{1},\cdots,\)\(c_{V}\) for \(V\leq M\). Figure 2 shows an example of the sorting process. In state \(s_{t}\), knapsacks are sorted as \(a\), \(b\), \(c\), \(d\) according to the initial capacities. The capacities are denoted by \(capa_{a}\), \(capa_{b}\), \(capa_{c}\), and \(capa_{d}\). When an item corresponding to the \(action_{t}\) is selected and the weight constraint condition is satisfied, \(capa_{a}\) changes from the existing capacity to a value reduced by the weight of the item. Assume that the currently changed capacity of knapsack \(a\) is smaller than \(capa_{b}\) and larger than \(capa_{c}\) and \(capa_{d}\). At this point, the order of the knapsacks are sorted so that the subsequent state \(s_{t+1}\) would contain the knapsacks' current capacities that are \(c_{1}=capa_{b}\), \(c_{2}=capa_{a}^{\prime}\), \(c_{3}=capa_{c}\), and \(c_{4}=capa_{d}\). For the following action in state \(s_{t+1}\), the capacity of \(b\) changes, and knapsacks for \(c_{j}\) are sorted again. When \(capa_{a}^{\prime\prime}\) becomes zero in \(s_{t+2}\), knapsack \(a\) becomes invalid.