Question: I have two very large NumPy arrays containing coordinates [[x1,y1,z1],[x2,y2,z2]...] (~10^9 elements) The arrays are of different sizes, and there will be overlap between the coordinates. So [x1,y1,z1] may be in array1, but not in array2. I would like to quickly get all of the coordinates that are in one, and only one array. For example, import numpy as np array1 = np.array([[1,2,3],[2,3,4],[3,4,5],[4,5,6]]) array2 = np.array([[2,3,4],[3,4,5],[4,5,6],[5,6,7]]) array_diff = some_function_to_get_diff(array1,array2)  would get: array_diff: np.array([[1,2,3],[5,6,7]])  I can use setdiff1d for one dimension, but it isn't clear to me how to do this for a 2d array. The files are quite large – 10GB or so, so being able to do this quickly/chunked in parallel would be a big plus. Update def some_faster_function_to_get_diff(array1, array2): unique1 = np.unique(array1, axis=0) # Get the unique coordinates in array1 unique2 = np.unique(array2, axis=0) # Get the unique coordinates in array2 set1 = set(map(tuple, unique1)) set2 = set(map(tuple, unique2)) array1_only = np.array(list(set1 - set2)) array2_only = np.array(list(set2 - set1)) if array1_only.size == 0: return array2_only if array2_only.size == 0: return array1_only return np.vstack((array1_only, array2_only)) def symmetric_difference(arr1, arr2): dtype = [('f{}'.format(i), arr1.dtype) for i in range(arr1.shape[1])] struct_arr1 = np.ascontiguousarray(arr1).view(dtype) struct_arr2 = np.ascontiguousarray(arr2).view(dtype) unique_to_arr1 = np.setdiff1d(struct_arr1, struct_arr2, assume_unique=True) unique_to_arr2 = np.setdiff1d(struct_arr2, struct_arr1,assume_unique=True) result = np.concatenate([unique_to_arr1, unique_to_arr2]).view(arr1.dtype).reshape(-1, arr1.shape[1]) return result from sys import getsizeof for i in range(2,9): no_ele=3*10**i arr1 = np.random.rand(no_ele, 3) * 100 arr2 = np.random.rand(no_ele, 3) * 100 print(f"i={i}: memory {getsizeof(arr2)/1024**3:2f}GB") arr2[50:60] = arr1[50:60] t0=time.time() diff1=symmetric_difference(arr1,arr2) t1=time.time() print(f"{no_ele}: contiguous {t1-t0}") t0=time.time() diff2=some_faster_function_to_get_diff(arr1,arr2) t1=time.time() print(f"{no_ele}: sets {t1-t0}") Gives: 2 memory 0.000007GB 300: contiguous 0.0004420280456542969 300: sets 0.0007300376892089844 3 memory 0.000067GB 3000: contiguous 0.0038480758666992188 3000: sets 0.0068819522857666016 4 memory 0.000671GB 30000: contiguous 0.052056074142456055 30000: sets 0.07555103302001953 5 memory 0.006706GB 300000: contiguous 0.6442217826843262 300000: sets 0.9859399795532227 6 memory 0.067055GB 3000000: contiguous 9.833515882492065 3000000: sets 12.196370840072632 7 memory 0.670552GB 30000000: contiguous 142.16755604743958 30000000: sets 138.87937593460083  @Andrés code works, and is much quicker than using lists, converting to contiguous array also produces a similar run time. The problem now is that the arrays are too big to fit into memory - helpfully the package (laspy) allows chunking the data like: with laspy.open(las_file1) as f1, laspy.open(las_file2) as f2: for points1 in f1.chunk_iterator(chunk_size):

Answer: For simple arrays there is setdiff1d() in numpy, however, this does not work for your arrays. If you would write your function from scratch, you could follow one of the following paths: Using list comprehension: def some_function_to_get_diff(array1, array2): unique1 = np.unique(array1, axis=0) # Get the unique coordinates in array1 unique2 = np.unique(array2, axis=0) # Get the unique coordinates in array2 # Get the coordinates that are in only one array array1_only = np.array([x for x in unique1.tolist() if x not in unique2.tolist()]) array2_only = np.array([x for x in unique2.tolist() if x not in unique1.tolist()]) # Special cases with one empty array if array1_only.size == 0: return array2_only if array2_only.size == 0: return array1_only return np.vstack((array1_only, array2_only))  Needs ~53s on two arrays with 105 elements each on my computer.  Using set operations: def some_faster_function_to_get_diff(array1, array2): unique1 = np.unique(array1, axis=0) # Get the unique coordinates in array1 unique2 = np.unique(array2, axis=0) # Get the unique coordinates in array2 set1 = set(map(tuple, unique1)) set2 = set(map(tuple, unique2)) array1_only = np.array(list(set1 - set2)) array2_only = np.array(list(set2 - set1)) if array1_only.size == 0: return array2_only if array2_only.size == 0: return array1_only return np.vstack((array1_only, array2_only))  Needs ~0.2s on two arrays with 105 elements each on my computer.