Solving Combinatorial Problems with PySpark | by Salil Jain | Towards Data Science

Let us consider the problem statement. Given n real numbers x1, x2, xn, choose any set of distinct numbers such that function f on those chosen numbers gives maximum value. function f can take any number of inputs so one can choose any number of numbers.
Expected output: A set of numbers S = {xi, xj,..., xm} or the maximum value for f(xi, xj,..., xm).
A brute force strategy for solving this problem would be to permute over the numbers so first, one selects 1 out of n numbers, then 2, then 3, and so on. If we don’t know function f then we wouldn’t know if the order of the input parameters is important or not. Hence we need permutation vs combination. If the order is not important then we can use the combination.
The number of k-combinations for all k is the number of subsets of a set of n elements. There are several ways to see that this number is 2^n.
Source: https://en.wikipedia.org/wiki/Combination
It is easy to see if there are n numbers possible way to select different k numbers for all k is very close to the binary representation of 2^n. An n bit binary vector can represent numbers from 0 to 2^n-1. If each index of this vector represents one of the n numbers, we can select that number if the bit for that index is on. This way we can iterate through numbers from 0 to 2^n-1, get their representation as a binary vector of n bit, and then select the numbers based on which bits are on.
Method get_data converts num into binary representation and based on which bits are on that index from data is returned.
Output:000 []001 [1]010 [4]011 [4, 1]100 [3]101 [3, 1]110 [3, 4]111 [3, 4, 1]
As one can see from the above output, a binary string 010 will get the value 4 at index 1 while 011 will get [4, 1] at index 1 and 2. Once we have the values, we can easily calculate the value of function f and eventually maximum.
Now to parallelize the calculation of function f over various combinations from the dataset, we would like to divide the number of all combinations 2^n in equal proportion.
The above methods divide the number counts in an equal number of buckets (division). num_per_division is the number ina division and spill is the remainder which is added to initial divisions until it becomes zero. The above method is used to divide 2^n in the number of buckets and then computations of function f are done on each bucket parallelly.
process_combination method processes one set of numbers computed from equal divisions. It executes the function f over all those numbers and returns the maximum value.
solve_combination puts everything together. First, it computes the number of combinations and then divides them into equal buckets. Then parallelize those buckets to compute function f over all buckets using the process_combination method.
There are various cases, where knowledge of the function f can reduce the search space drastically.
Let's say function f calculates the mean of all input parameters (numbers).
If the highest number is chosen from all the input data, the mean of that would be highest. So in this scenario, we can just choose the highest number, rather than using spark.
Similar to mean if function f computes the sum of all input parameters (numbers).
We can just sum all the numbers from input data because that would be the highest.
If function f computes the sine of the sum of all the input parameters then we would rather use the spark solution. There could be a more complicated function f where the spark solution could be immensely helpful.
Above I discussed how mean and sum could be easily calculated without the spark solution. I use the same paradigm to do unit testing. In the case of mean, I compare the output from spark to the maximum value of input data. And in the case of the sum, I compare the spark output to the sum of all input data.
In this post, I laid out a simple framework on how to solve the combinatorial problem using a simple function f, though the solution is not restricted to that and could be applied to a myriad set of problems. Hope you enjoyed it!