masks: avoid graphical glitches at maximum zoom level. fixes #9867