refine an object mask
(eroding a table mask for the prism, removing small areas in an object mask)