Question: Hey guys I have a script that compares each possible user and checks how similar their text is:  dictionary = { t.id: ( t.text, t.set, t.compare_string ) for t in dataframe.itertuples() } highly_similar = [] for a, b in itertools.combinations(dictionary.items(), 2): if a[1][2] == b[1][2] and not a[1][1].isdisjoint(b[1][1]): similarity_score = fuzz.ratio(a[1][0], b[1][0]) if (similarity_score >= 95 and len(a[1][0]) >= 10) or similarity_score == 100: highly_similar.append([a[0], b[0], a[1][0], b[1][0], similarity_score])  This script takes around 15 minutes to run, the dataframe contains 120k users, so comparing each possible combination takes quite a bit of time, if I just write pass on the for loop it takes 2 minutes to loop through all values. I tried using filter() and map() for the if statements and fuzzy score but the performance was worse. I tried improving the script as much as I could but I don't know how I can improve this further. Would really appreciate some help!

Answer: It is slightly complicated to reason about the data since you have not attached it, but we can see multiple places that might provide an improvement:  First, let's rewrite the code in a way which is easier to reason about than using the indices: dictionary = { t.id: ( t.text, t.set, t.compare_string ) for t in dataframe.itertuples() } highly_similar = [] for a, b in itertools.combinations(dictionary.items(), 2): a_id, (a_text, a_set, a_compre_string) = a b_id, (b_text, b_set, b_compre_string) = b if (a_compre_string == b_compre_string and not a_set.isdisjoint(b_set)): similarity_score = fuzz.ratio(a_text, b_text) if (similarity_score >= 95 and len(a_text) >= 10) or similarity_score == 100): highly_similar.append( [a_id, b_id, a_text, b_text, similarity_score])   You seem to only care about pairs having the same compare_string values. Therefore, and assuming this is not something that all pairs share, we can key by whatever that value is to cover much less pairs. To put some numbers into it, let's say you have 120K inputs, and 1K values for each value of val[1][2] - then instead of covering 120K * 120K = 14 * 10^9 combinations, you would have 120 bins of size 1K (where in each bin we'd need to check all pairs) = 120 * 1K * 1K = 120 * 10^6 which is about 1000 times faster. And it would be even faster if each bin has less than 1K elements. import collections # Create a dictionary from compare_string to all items # with the same compare_string items_by_compare_string = collections.defaultdict(list) for item in dictionary.items(): compare_string = item[1][2] items_by_compare_string[compare_string].append(items) # Iterate over each group of items that have the same # compare string for item_group in items_by_compare_string.values(): # Check pairs only within that group for a, b in itertools.combinations(item_group, 2): # No need to compare the compare_strings! if not a_set.isdisjoint(b_set): similarity_score = fuzz.ratio(a_text, b_text) if (similarity_score >= 95 and len(a_text) >= 10) or similarity_score == 100): highly_similar.append( [a_id, b_id, a_text, b_text, similarity_score])   But, what if we want more speed? Let's look at the remaining operations:  We have a check to find if two sets share at least one item  This seems like an obvious candidate for optimization if we have any knowledge about these sets (to allow us to determine which pairs are even relevant to compare) Without additional knowledge, and just looking at every two pairs and trying to speed this up, I doubt we can do much - this is probably highly optimized using internal details of Python sets, I don't think it's likely to optimize it further   We a fuzz.ratio computation which is some external function, and I'm going to assume is heavy  If you are using this from the FuzzyWuzzy package, make sure to install python-Levenshtein to get the speedups detailed here   We have some comparisons which we are unlikely to be able to speed up  We might be able to cache the length of a_text by nesting the two loops, but that's negligible   We have appends to a list, which runs on average ("amortized") constant time per operation, so we can't really speed that up    Therefore, I don't think we can reasonably suggest any more speedups without additional knowledge. If we know something about the sets that can help optimize which pairs are relevant we might be able to speed things up further, but I think this is about it. EDIT: As pointed out in other answers, you can obviously run the code in multi-threading. I assumed you were looking for an algorithmic change that would possibly reduce the number of operations significantly, instead of just splitting these over more CPUs.