ENH: respect dtype in masking