Improve handling of NaN/NA/Inf/-Inf in doubles