Improve memory usage when checking for NaNs