brush masks: better ui control during creation
brush masks: fix bug in Ramer-Douglas-Peucker algorithm