## 1 Introduction  
A knapsack problem is one of combinatorial optimization problems. Extending the single knapsack problem, the multiple knapsack problem (MKP) is a problem that finds disjoint subsets to maximize the total profit in knapsacks. In a nutshell, each subset can be distributed to different knapsacks, as long as the total weight of each subset is equal to or less than the capacity of the corresponding knapsack. MKP is formulated as follows:  
\[\max_{x}\sum_{j}\sum_{i}p_{i}x_{ij} \tag{1}\] \[\text{subject to}\sum_{i}w_{i}x_{ij}\leq c_{j},\ \ \sum_{j}x_{ij}\leq 1\] \[\text{for}\ \ \forall i\in I,\forall j\in J,\ \ x_{ij}\in\{0,1\}\]  
where \(J\) is the pool of knapsacks, \(I\) is the pool of items, \(p_{i}\) is the profit of the \(i\)-th item, \(w_{i}\) is the weight of the \(i\)-th item, and \(x_{ij}\) is a decision variable to indicate whether the \(i\)-th item is selected to be put in the \(j\)-th knapsack. The exact method has the computational complexity of \(O((J+1)^{|I|})\). MKP arises in substantial real-world cases like vehicle/container loading, production scheduling, and resource allocation in the computer network systems [1, 2]. [PERSON] et al. [2] formulated production scheduling as MKP. It deals with assigning items to a period of the production schedule to maximize the profit of production and minimize the cost keeping the constraint of capacities. [PERSON] et al. [3] formulated schedulingappliances as MKP in smart grid infrastructure. To minimize electricity bills in a household, it considers an appliance as an item, its energy consumption as weight, and the time period as a knapsack. [PERSON] et al. [4] formulated a multi-user computation offloading problem as MKP. To maximize the profit of user equipment, it considers user equipment as items, requested CPU as weights, and CPU capacities of mobile edge computing server as capacities of knapsacks. [PERSON] et al. [5] formulated Virtual Network Functions (VNF) placement problems as MKPs. It considers a data center as a knapsack, a service request as an item, and the quantity of the service requests as weight. Then, it solves the problem to maximize the total priority level of the requests. In various application fields, solving MKP with high performance in practical time to maximize revenue is unavoidable. In MKP, there is a trade-off between empirical performance and reduced computational complexity. Note that MKP is an NP-hard problem [6]. Traditional approaches, such as heuristic or genetic algorithms, focus on reducing their computational complexity. These days, the development of deep neural network (DNN) accelerates the application of Machine Learning (ML) to various engineering problems. Reinforcement Learning (RL), one type of ML, is an experience-driven method that makes autonomous agents solve decision-making problems [7]. Deep reinforcement learning (DRL) has been applied not only to robot movement [8] and games such as AlphaGo [9] but also to discrete combinatorial optimization problems [10; 11; 12; 13; 14; 15]. DRL agents trained with DNN can solve the high dimensional problems in a reasonable amount of time [2; 16; 17]. Recently, traveling salesman problem (TSP) and knapsack problem have been solved not only with supervised learning [10; 18], but also with RL [11; 19]. DRL with experience replay can be applied to the MKP problem because it can deal with various combinations of items and knapsacks in real-time without predefined massive data for training. In the literature, there are some approaches that solve a single knapsack problem, but there are lack of approaches for MKPs [10; 11; 19]. In this paper, we propose a DRL-based method to solve MKP. The proposed DRL method solves MKP by exploiting an existing Markov Decision Process (MDP) model for single knapsack problem in [11] and devising a novel capacity-aware knapsack selection method for MKP. Each state represents a separate problem state, and the agent's action decides which item to be mapped to a knapsack with the greatest capacity.