Finding Distant Pairs in Python with Pandas | by Chris Morrow | Towards Data Science

An occasional problem that comes up in Computer Science and Data Science is the need to find a pair of numbers in a set of unique numbers that are the farthest from one another. These pair of numbers are referred to as distant pairs, or max distant pairs.
Thankfully, with Python and Pandas, we can find the distant pairs in a set of unique values relatively easily. Let’s start by importing NumPy and Pandas which will be used for generating and manipulating our data.
import numpy as npimport pandas as pd
Next, we’ll generate a random sample of 7 integers between 100000 and 999999 as one_dim .
np.random.seed(123)one_dim = np.random.randint(100000, 999999, size=7)one_dimarray([873630, 377869, 128030, 117730, 292476, 294278, 770737])
The np.random.seed(123) statement guarantees that the random number generator will produce reproducible values. The variable one_dim is a NumPy array with our 7 random integers that represent the set of one dimensional values to check for max distant pairs.
The distant pairs can be found using the following steps:
Take the cartesian product of the set with itself and apply the distance algorithm to every pair.Find the maximum value in the cartesian product set of distances.Determine which pairs in the original set produce the maximum distance.
Take the cartesian product of the set with itself and apply the distance algorithm to every pair.
Find the maximum value in the cartesian product set of distances.
Determine which pairs in the original set produce the maximum distance.
The cartesian product of the set with itself can be represented as a Pandas DataFrame, where the index and column headers of the DataFrame are the values in the set.
df = pd.DataFrame(index=test, columns=test)df+------+------+------+------+------+------+------+------+|      |873630|377869|128030|117730|292476|294278|770737|+------+------+------+------+------+------+------+------+|873630| NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||377869| NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||128030| NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||117730| NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||292476| NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||294278| NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||770737| NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  |+------+------+------+------+------+------+------+------+
This creates a n x n DataFrame, where n is the number of values in the set. Since this DataFrame was created without data, all the values are empty (NaN).
The next step is to apply the distance algorithm which is abs(a - b) where a and b are both values in the original set. Applying this algorithm to every cell in the DataFrame is equivalent to calculating the distance between every value in the original set.
We accomplish this by using the DataFrame object’s apply() method, where the argument is a callable function. Since the distance algorithm is simple, a lambda function will suffice. Using apply()is always faster than using nested for loops because apply() is a vectorized operation applied to every value in a column or row simultaneously. In essence, apply() is a O(n) time operation where n is proportional to the number of rows and nested for loops is a O(n2) time operation.
diff = df.apply(lambda col: abs(col.name - col.index))diff+------+------+------+------+------+------+------+------+|      |873630|377869|128030|117730|292476|294278|770737|+------+------+------+------+------+------+------+------+|873630|0     |495761|745600|755900|581154|579352|102893||377869|495761|0     |249839|260139|85393 |83591 |392868||128030|745600|249839|0     |10300 |164446|166248|642707||117730|755900|260139|10300 |0     |174746|176548|653007||292476|581154|85393 |164446|174746|0     |1802  |478261||294278|579352|83591 |166248|176548|1802  |0     |476459||770737|102893|392868|642707|653007|478261|476459|0     |+------+------+------+------+------+------+------+------+
What did this do? The apply() method performs the lambda function on each column and subtracts the name of the column (a scalar value) from the index of the column (a vector).
This populates the DataFrame with the absolute value distance between each pair in the original set. Notice how all the values along the diagonal are 0, because a number subtracted from itself is 0. This also indicates that the DataFrame is partitioned into two sections, each a triangle that mirror’s the other. This means the max distance is in the DataFrame twice, once per triangle.
The next step is to find the max distance. This is done by calling the DataFrame’s max() method, first on each column, and then on each value of the resulting Series.
max = diff.max().max()755900
Now that we know the max distance, the last step is to determine which two pairs in the one dimensional set form the max distance. Since the index and column headers of the DataFrame are the values in the one dimensional set, the DataFrame can be filtered to just the max distance value.
Because the max distance value appears in the DataFrame twice, filtering the DataFrame will produce a 2 x 2 DataFrame where both values in the index and column headers are the distant pairs.
filtered = (    diff[diff == max]    .dropna(how='all')    .dropna(how='all', axis=1))filtered+------+------+------+|      |873630|117730|+------+------+------+|873630| NaN  |755900||117730|755900| NaN  |+------+------+------+
Finally, getting the distant pairs is as simple as getting the index of the filtered DataFrame.
pairs = filtered.index.to_list()pairs[873630, 117730]
This works well and is a pretty efficient solution that scales, but only works for sets of one dimensional values. What if we wanted to apply this method to a set of two dimensional values? It turns out it’s not that different and only requires minor tweaks. Let’s start by generating a new set of two dimensional values.
two_dim = np.random.randint(100, size=(7, 2))two_dimarray([[83, 57],       [86, 97],       [96, 47],       [73, 32],       [46, 96],       [25, 83],       [78, 36]])
The first difference compared to the one dimensional case comes from the fact that a DataFrame index must be one dimensional. We can get around this by creating a reference DataFrame that represents our set of two dimensional values, and use the index of this reference DataFrame in the cartesian product instead.
ref = pd.DataFrame.from_records(two_dim, columns=('x', 'y'))ref+----+----+----+|    | x  | y  |+----+----+----+| 0  | 83 | 57 || 1  | 86 | 97 || 2  | 96 | 47 || 3  | 73 | 32 || 4  | 46 | 96 || 5  | 25 | 83 || 6  | 78 | 36 |+----+----+----+
Next, we’ll create the cartesian product DataFrame with the index of the reference DataFrame.
df2 = pd.DataFrame(index=ref.index, columns=ref.index)df2+------+------+------+------+------+------+------+------+|      |0     |1     |2     |3     |4     |5     |6     |+------+------+------+------+------+------+------+------+|0     | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||1     | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||2     | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||3     | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||4     | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||5     | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  ||6     | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  | NaN  |+------+------+------+------+------+------+------+------+
The second difference compared to the one dimensional case is the lambda function needs to be swapped out for a two dimensional distance algorithm. This is just the pythagorean theorem.
diff2 = df2.apply(    lambda col: np.sqrt(        abs(ref.iloc[col.name]['x'] - ref.iloc[col.index]['x']) ** 2 +        abs(ref.iloc[col.name]['y'] - ref.iloc[col.index]['y']) ** 2    ))diff2+------+------+------+------+------+------+------+------+|      |0     |1     |2     |3     |4     |5     |6     |+------+------+------+------+------+------+------+------+|0     |0.0000|40.112|16.401|26.926|53.759|63.561|21.587||1     |40.112|0.0000|50.990|66.287|40.012|62.586|61.522||2     |16.401|50.990|0.0000|27.459|70.007|79.605|21.095||3     |26.926|66.287|27.459|0.0000|69.462|70.036|6.4031||4     |53.759|40.012|70.007|69.462|0.0000|24.698|68.000||5     |63.561|62.586|79.605|70.036|24.698|0.0000|70.838||6     |21.587|61.522|21.095|6.4031|68.000|70.838|0.0000|+------+------+------+------+------+------+------+------+
This requires some explanation. The lambda function here is the pythagorean theorem: c = √(abs(a)2 + abs(b)2), but the x and y values from the reference DataFrame are fetched by their index value — ref.iloc[col.name]['x'] (scalar value) and ref.iloc[col.index]['x'] (vector).
Next, we’ll get the max value, same as before.
max = diff2.max().max()max79.605
Getting the distant pairs works the same way, but the index value of the cartesian product DataFrame need to be swapped out for the corresponding x and y value in the reference DataFrame.
filtered2 = (    diff2[diff2 == max]    .dropna(how='all')    .dropna(how='all', axis=1))filtered2+------+------+------+|      |2     |5     |+------+------+------+|2     | NaN  |79.605||5     |79.605| NaN  |+------+------+------+
Finally, get the distant pairs.
pairs = [ref.iloc[idx].to_list() for idx in filtered2.index]pairs[[96, 47], [25, 83]]
And that’s it!