R600: Compute masked bits for min and max