You're still using two copies of the image and making that copy inside of the inner loop. Casting here (instead of outside the loop) is probably slowing things down. Is there a reason for not using a single dtype for all operations?