Genetic Algorithm in R: The Knapsack Problem | by Raden Aurelius Andhika Viadinugroho | Towards Data Science

Motivation
The Knapsack Problem is one of the famous problems in optimization, specifically in combinatoric optimization. The motivation of this problem comes where someone needs to maximize his knapsack capacity — hence the name — with the most valuable items possible. There are many approaches to solve this problem, but in this article, I will give you an example to solve this problem using the Genetic Algorithm approach in R.
The Knapsack Problem
In this article, the knapsack problem that we will try to solve is the 0–1 knapsack problem. Given a set of n items numbered from 1 to n, each with weight w_i and a value v_i. Suppose that each item copies are restricted to 1, i.e. the item is either included in the knapsack or not. Here we want to maximize the objective function (i.e. the values of items in the knapsack)
where the objective function is subject to the constraint function
Genetic Algorithm Concepts
Genetic Algorithm is one of the optimization algorithms based on the evolution concept by natural selection. As proposed by Charles Darwin, evolution by natural selection is the mechanism on how many varieties of living things will adapt to the environment to survive through two principles: natural selection and mutation. Based on that concept, the Genetic Algorithm's goal is to gain the optimal solutions of the objective function by selecting the best or fittest solution alongside the rare and random mutation occurrence. The algorithm itself can be explained as follows.
Initialize the data and/or the function that we will optimize.Initialize the population size, maximum iteration number (the number of generations), crossover probability, mutation probability, and the number of elitism (the best or fittest individual that won’t undergo mutation).From the population, select two individuals, then perform crossover between two individuals with probability p.Then, perform mutation between two individuals with probability p (usually the probability of mutation are really low).Repeat steps 3 and 4 until all individuals in one generation are trained. These all individuals will be used for training the next generation until the number of generations is reaching the limit.
Initialize the data and/or the function that we will optimize.
Initialize the population size, maximum iteration number (the number of generations), crossover probability, mutation probability, and the number of elitism (the best or fittest individual that won’t undergo mutation).
From the population, select two individuals, then perform crossover between two individuals with probability p.
Then, perform mutation between two individuals with probability p (usually the probability of mutation are really low).
Repeat steps 3 and 4 until all individuals in one generation are trained. These all individuals will be used for training the next generation until the number of generations is reaching the limit.
To give you a better understanding of the genetic algorithm, let’s jump to the case study where we will use the genetic algorithm to solve the knapsack problem in R.
Case Study: Solving Knapsack Problem using Genetic Algorithm
Suppose that you want to go hiking with your friends, and you have the items that you can use for hiking, with the weight (in kilograms) and survival point of each item, respectively, as follows.
Suppose too that you have a knapsack that can contain items with a maximum capacity of 25 kilograms, where you can only bring one copy for each item. The goal is: you want to maximize your knapsack capacity while maximizing your survival points as well. From the problem statement, we can define the weight of the items as the constraint function, while the survival points cumulated from the items in the knapsack as the objective function.
For the implementation in this article, here we using the GA library in R created by Luca Scrucca [2]. First, we need to input the data and the parameters that we used by writing these lines of code.
#0-1 Knapsack's Problemlibrary(GA)item=c('raincoat','pocket knife','mineral water','gloves','sleeping bag','tent','portable stove','canned food','snacks')weight=c(2,1,6,1,4,9,5,8,3)survival=c(5,3,15,5,6,18,8,20,8)data=data.frame(item,weight,survival)max_weight=25
To give you a better understanding of the genetic algorithm in this problem, suppose that we only bring a pocket knife, mineral water, and snacks in your knapsack, initially. We can write it as the “chromosome”, and since the problem that we want to solve is a 0–1 knapsack problem, then from these lines of codes below, 1 means that we bring the item, while 0 means that we left the item.
#1 means that we bring the item, while 0 means that we left the itemchromosomes=c(0,1,1,0,0,0,0,0,1)data[chromosomes==1,]
We can see the result as follows.
> data[chromosomes==1,]           item weight survival2  pocket knife      1        33 mineral water      6       159        snacks      3        8
Then, we create the objective function that we want to optimize with the constraint function by writing these lines of code below. The fitness the function will we use in the ga function from the GA library.
#create the function that we want to optimizefitness=function(x){  current_survpoint=x%*%data$survival  current_weight=x%*%data$weight  if(current_weight>max_weight)  {    return(0)  }  else  {    return(current_survpoint)  }}
Now here’s the interesting part: the optimization process using the genetic algorithm. Suppose that we want to create a maximum of 30 generations and 50 individuals for the optimization process. For reproducibility, we write the seed argument and keep the best solution.
GA=ga(type='binary',fitness=fitness,nBits=nrow(data),maxiter=30,popSize=50,seed=1234,keepBest=TRUE)summary(GA)plot(GA)
By running the line of code above, we can achieve the result as follows.
> GA=ga(type='binary',fitness=fitness,nBits=nrow(data),maxiter=30,popSize=50,seed=1234,keepBest=TRUE)GA | iter = 1 | Mean = 31.92 | Best = 61.00GA | iter = 2 | Mean = 31.32 | Best = 61.00GA | iter = 3 | Mean = 33.08 | Best = 61.00GA | iter = 4 | Mean = 36.14 | Best = 61.00GA | iter = 5 | Mean = 42.42 | Best = 61.00GA | iter = 6 | Mean = 36.56 | Best = 61.00GA | iter = 7 | Mean = 37.32 | Best = 61.00GA | iter = 8 | Mean = 38.18 | Best = 61.00GA | iter = 9 | Mean = 39.02 | Best = 61.00GA | iter = 10 | Mean = 38.92 | Best = 61.00GA | iter = 11 | Mean = 37.54 | Best = 61.00GA | iter = 12 | Mean = 35.14 | Best = 61.00GA | iter = 13 | Mean = 36.28 | Best = 61.00GA | iter = 14 | Mean = 40.82 | Best = 61.00GA | iter = 15 | Mean = 44.26 | Best = 61.00GA | iter = 16 | Mean = 41.62 | Best = 61.00GA | iter = 17 | Mean = 38.66 | Best = 61.00GA | iter = 18 | Mean = 36.24 | Best = 61.00GA | iter = 19 | Mean = 43 | Best = 61GA | iter = 20 | Mean = 43.48 | Best = 61.00GA | iter = 21 | Mean = 43.08 | Best = 61.00GA | iter = 22 | Mean = 44.88 | Best = 61.00GA | iter = 23 | Mean = 46.84 | Best = 61.00GA | iter = 24 | Mean = 46.8 | Best = 61.0GA | iter = 25 | Mean = 42.62 | Best = 61.00GA | iter = 26 | Mean = 46.52 | Best = 61.00GA | iter = 27 | Mean = 46.14 | Best = 61.00GA | iter = 28 | Mean = 43.8 | Best = 61.0GA | iter = 29 | Mean = 46.16 | Best = 61.00GA | iter = 30 | Mean = 42.6 | Best = 61.0> summary(GA)-- Genetic Algorithm -------------------   GA settings: Type                  =  binary Population size       =  50 Number of generations =  30 Elitism               =  2 Crossover probability =  0.8 Mutation probability  =  0.1GA results: Iterations             = 30 Fitness function value = 61 Solution =      x1 x2 x3 x4 x5 x6 x7 x8 x9[1,]  1  0  1  1  0  0  1  1  1> GA@summary     max  mean q3 median q1 min[1,]  61 31.92 48   36.0 16   0[2,]  61 31.32 47   31.0 24   0[3,]  61 33.08 51   36.5 13   0[4,]  61 36.14 52   39.0 31   0[5,]  61 42.42 56   47.5 38   0[6,]  61 36.56 52   41.5 26   0[7,]  61 37.32 54   43.0 29   0[8,]  61 38.18 54   43.0 29   0[9,]  61 39.02 55   47.0 33   0[10,]  61 38.92 52   43.5 33   0[11,]  61 37.54 48   39.0 33   0[12,]  61 35.14 47   39.0 29   0[13,]  61 36.28 47   41.0 23   0[14,]  61 40.82 51   43.0 34   0[15,]  61 44.26 51   48.0 38  20[16,]  61 41.62 52   45.0 34   0[17,]  61 38.66 53   41.5 28   0[18,]  61 36.24 51   39.5 28   0[19,]  61 43.00 56   48.0 37   0[20,]  61 43.48 56   48.0 39   0[21,]  61 43.08 56   48.0 40   0[22,]  61 44.88 56   51.0 41   0[23,]  61 46.84 56   52.0 41   0[24,]  61 46.80 56   48.0 41   0[25,]  61 42.62 56   48.0 33   0[26,]  61 46.52 56   48.0 42   0[27,]  61 46.14 54   47.0 43   0[28,]  61 43.80 56   49.5 40   0[29,]  61 46.16 54   50.0 43   0[30,]  61 42.60 56   48.0 36   0
From the result above, we can see that the performance for every individual is increasing in each generation. We can see it from the fitness value (i.e. the survival points in this case) mean and median that tends to increases in each generation. Let’s try to train it again, but with more generations.
GA2=ga(type='binary',fitness=fitness,nBits=nrow(data),maxiter=50,popSize=50,seed=1234,keepBest=TRUE)GA3=ga(type='binary',fitness=fitness,nBits=nrow(data),maxiter=100,popSize=50,seed=1234,keepBest=TRUE)plot(GA2)plot(GA3)
From GA2 and GA3 , we can see that the optimization result for each individual is at their best on generation 40-ish and 60-ish, according to the mean and median of fitness value on that generation. We can also see that the best fitness value is increasing to 62 from 72nd generation onwards.
Since we keep the best result on every optimization process, we want to find out the items that we can bring for hiking based on the best result from the genetic algorithm optimization. We can see the summary from GA3 as follows.
> summary(GA3)-- Genetic Algorithm -------------------   GA settings: Type                  =  binary Population size       =  50 Number of generations =  100 Elitism               =  2 Crossover probability =  0.8 Mutation probability  =  0.1GA results: Iterations             = 100 Fitness function value = 62 Solution =      x1 x2 x3 x4 x5 x6 x7 x8 x9[1,]  1  1  1  1  1  0  0  1  1
From the result above, we can conclude that the items that we can include in the knapsack are a raincoat, pocket knife, mineral water, gloves, sleeping bag, canned food, and snacks. We can calculate the weight of the knapsack, to make sure that the knapsack is not over capacitated.
> chromosomes_final=c(1,1,1,1,1,0,0,1,1)> cat(chromosomes_final%*%data$weight)25
Great! We can see that the weight of the items is the same as the knapsack capacity!
Conclusion
And there you go! You can go hiking with your friends (best to do after the pandemic is over, of course) with maximum survival points and capacity by implementing the genetic algorithm in R. In fact, you can solve the knapsack problem by using genetic algorithm in many real-world applications, such as choosing the best-performing portfolio, production scheduling, and many more.
As usual, feel free to contact me via my LinkedIn if you have any questions or discussions. Stay safe and stay healthy!
References
[1] G. B. Matthews, On the Partition of Numbers (1897), Proceedings of the London Mathematical Society.
[2] Luca Scrucca, GA: A Package for Genetic Algorithms in R (2013), Journal of Statistical Software.
[3] https://www.rdocumentation.org/packages/GA/versions/3.2/topics/ga
[4] Harvey M. Salkin and Cornelis A. De Kluyver, The knapsack problem: A survey (1975), Naval Research Logistics Quarterly.
[5] Krzysztof Dudziński and Stanisław Walukiewicz, Exact methods for the knapsack problem and its generalizations (1987), European Journal of Operational Research.
[6] Sami Khuri, Thomas Bäck, and Jörg Heitkötter, The zero/one multiple knapsack problem and genetic algorithms (1994), SAC ’94: Proceedings of the 1994 ACM symposium on Applied computing.