I still would expect that a Cython implementation could be faster. In particular, you don't have to store all the values of the patches SSD, only the smallest one (and the position of the corresponding point).