Question: I have a dataframe with a lot of rows with numerical columns, such as:     A B C D     12 7 1 0   7 1 2 0   1 1 1 1   2 2 0 0     I need to reduce the size of the dataframe by removing those rows that has another row with all values bigger. In the previous example i need to remove the last row because the first row has all values bigger (in case of dubplicate rows i need to keep one of them). And return This:     A B C D     12 7 1 0   7 1 2 0   1 1 1 1     My faster solution are the folowing:  def complete_reduction(df, columns): def _single_reduction(row): df["check"] = True for col in columns: df["check"] = df["check"] & (df[col] >= row[col]) drop_index.append(df["check"].sum() == 1) df = df.drop_duplicates(subset=columns) drop_index = [] df.apply(lambda x: _single_reduction(x), axis=1) df = df[numpy.array(drop_index).astype(bool)] return df  Any better ideas?  Update: A new solution has been found here https://stackoverflow.com/a/68528943/11327160 but i hope for somethings faster.

Answer: An more memory-efficient and faster solution than the one proposed so far is to use Numba. There is no need to create huge temporary array with Numba. Moreover, it is easy to write a parallel implementation that makes use of all CPU cores. Here is the implementation: import numba as nb @nb.njit def is_dominated(arr, k): n, m = arr.shape for i in range(n): if i != k: dominated = True for j in range(m): if arr[i, j] < arr[k, j]: dominated = False if dominated: return True return False # Precompile the function to native code for the most common types @nb.njit(['(i4[:,::1],)', '(i8[:,::1],)'], parallel=True, cache=True) def dominated_rows(arr): n, m = arr.shape toRemove = np.empty(n, dtype=np.bool_) for i in nb.prange(n): toRemove[i] = is_dominated(arr, i) return toRemove # Special case df2 = df.drop_duplicates() # Main computation result = df2[~dominated_rows(np.ascontiguousarray(df.values))]   Benchmark The input test is two random dataframes of shape 20000x5 and 5000x100 containing small integers (ie. [0;100[). Tests have been done on a (6-core) i5-9600KF processor with 16 GiB of RAM on Windows. The version of @BingWang is the updated one of the 2022-05-24. Here are performance results of the proposed approaches so far: Dataframe with shape 5000x100 - Initial code: 114_340 ms - BENY: 2_716 ms (consume few GiB of RAM) - Bing Wang: 2_619 ms - Numba: 303 ms <---- Dataframe with shape 20000x5 - Initial code: (too long) - BENY: 8.775 ms (consume few GiB of RAM) - Bing Wang: 578 ms - Numba: 21 ms <----  This solution is respectively about 9 to 28 times faster than the fastest one (of @BingWang). It also has the benefit of consuming far less memory. Indeed, the @BENY implementation consume few GiB of RAM while this one (and the one of @BingWang) only consumes no more than few MiB for this used-case. The speed gain over the @BingWang implementation is due to the early stop, parallelism and the native execution. One can see that this Numba implementation and the one of @BingWang are quite efficient when the number of column is small. This makes sense for the @BingWang since the complexity should be O(N(logN)^(d-2)) where d is the number of columns. As for Numba, it is significantly faster because most rows are dominated on the second random dataset causing the early stop to be very effective in practice. I think the @BingWang algorithm might be faster when most rows are not dominated. However, this case should be very uncommon on dataframes with few columns and a lot of rows (at least, clearly on uniformly random ones).