Fix an overflow in setting xd3_source.maskby